An excellent point around data retention and training data. It's not like you can scrub out bits of the neural network weights. I guess such a product would have to factor this in.
Granted you'd have to test it without recording that, and you'd never really know for sure if you'd fully scrubbed it, only that you hadn't found a trace of it with your subsequent queries. ...now that I say that I'm sure there's a film about roughly this, delving into an AI trying to retrieve something hidden or something? The Wargames sequel perhaps?
Your point about fine-tuning it out is interesting though. As you say though, you'd have to be really sure you'd really erased it or made it "unreachable" .