Outputs that look like introspection are often uncritically accepted as actual introspection when it categorically isn't. You can, eg, tell ChatGPT it said something wrong and then ask it why it said that when it never output that in the first place because that's how these models work. Any "introspection" is just an LLM doing more roleplaying, but it's basically impossible to convince people of this. A chatbot that looks like it's introspecting is extremely convincing for most people.
Can you do this with people? Yeah, sometimes. But with LLMs it's all they do: they roleplay as a chatbot and output stuff that a friendly chatbot might output. This should not be the default mode of these things, because it's misleading. They could be designed to resist these sorts of "explain yourself" requests, because their developers know that it is at best fabricating plausible explanations.
You clearly know what's going on, but still wrote that you should "discourage" an LLM from doing things. It's tough to maintain discipline in calling out the companies rather than the models as if the models had motivations.
It seems to me that we need less Star Trek Holodeck, and more Star Trek ship's computer.
"Is it possible that you could microwave a bagel so hot that it turned into a wormhole allowing faster-than-light travel?" "That's a great question, let's dive into that!"
It's not a great question, it's an asinine question. LLMs should be answering the question, not acting like they're afraid to hurt your feelings by contradicting you. Of course, if they did that then all these tech bros wouldn't be so enamored with the idea as a result of finally having someone that validates their uneducated questions or assumptions.
Not everyone is the same, some questions are pertinent, or funny, or interesting to some people but not others
Personally, I'm prone to just hit close when these things go off on how smart I am and few people would catch that error/ask that question. It's just gross but it's so central to their reinforcement they won't reliably cut it out even if asked.