The last line of defense must not be AI

That’s fine in theory, but won’t fly in practice for all destructive, irreversible actions. As an example, how do you prevent a chatbot from generating a highly insulting/racist remark or incorrect or illegal advice that will, later cost you millions?

Human oversight is (deemed) too expensive.

A deterministic script can detect known profanities, but may suffer from a variant of the Scunthorpe problem (https://en.wikipedia.org/wiki/Scunthorpe_problem), and won’t detect unknown profanities or creative ones that don’t use any words that are considered profane. A deterministic script also is very bad at detecting legal issues with responses.

“Don’t reply a chatbot” will work for that, but for many, that doesn’t seem to be an option.