I find Meta’s approach to hallucinations delightfully counter intuitive. Basically they (and presumably OpenAI and others):

   - Extract a snippet of training data.
   - Generate a factual question about it using Llama 3.
   - Have Llama 3 generate an answer.
   - Score the response against the original data.
   - If incorrect, train the model to recognize and refuse incorrect responses.

In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer.

Instead of teaching the model to recognize what it doesn't know, why not teach it using those same examples? Of course the idea is to "connect the unused uncertainty neuron", which makes sense for out-of-context generalization. But we can at least appreciate why this wasn't an obvious thing to do for generation 1 LLMs.