The reward it gets from the reinforcement learning (RL) process probably didn’t include a sufficiently strong weight on being truthful.
Reward engineering for RL might be the most important area of research in AI now.
Reward engineering for RL might be the most important area of research in AI now.
Creating that training set though might cost many trillions of dollars though, since you need to basically recreate equivalent of internet but without any lies or bad intentions etc.
And I don't even think it's a matter of the LLM being malicious. Humans playing games get their reward from fun, and will naturally reset the game if the conditions do not lead to it.
I’d like to see better inference-time control of this behavior for sure; seems like a dial of some sort could be trained in.
A simple way to stop hallucinating would be to always state that "I don't know for sure, but my educated guess would be ..." but that's clearly not what we want.