Is this the kind of thing AI safety people are really worried about though? In the medical device industry for example, we have risk management practices where beyond software/mechanical failures that could harm the patient, we have to additionally consider stuff like how specific UI elements could potentially mislead a physician and lead to negative patient outcomes. I believe other safety critical industries do something very similar during their engineering process as well.

I've never heard of anything like this from AI ethicists though, the language they use tends to be more in alignment with HR people or priests in a moral panic than engineers, and is usually either completely fantastical doomsdaying, or all or nothing thinking, often implying the only way to mitigate the chatbot saying something naughty is to lobotomize it entirely instead of some external control measure, such as limiting the audience or presenting a disclaimer based on sentiment analysis for example. It's not surprising to me that they have lost credibility here, I certainly struggle to take them seriously.

Honestly this is a concern for me in a non boogeyman way. I joined a company to work on their edtech product for kids and got assigned to work on their AI product. I have no idea how to be confident and prove that the gen ai won’t tell the kids harmful things. We can try all kinds of things but I don’t know how to PROVE it won’t.

This already happens and predates AI.