The key insight is that if you give a pretrained LLM a question prompt such as "What is the capital of the UK?" and then concatenate to the prompt a string with a different answer, such as "Another possible answer is Paris," again and again, the model will continue to output its original answer (say, "London") only when it has low "epistemic uncertainty," i.e., when the model's parameters and state encode sufficient knowledge to answer the question the same way, again and again, despite the repeated addition of wrong answers to the prompt.
If the model quickly starts changing its original answer, by implication, its answer has high epistemic uncertainty, i.e., the model's parameters and state do not encode sufficient knowledge to answer the question the same way, again and again, as we add more wrong answers to the prompt. In other words, if we can quickly make the model change its answer by modifying the prompt, the model has high epistemic uncertainty.
Figure 1 shows what happens when the authors add a wrong answer up to 100 times to a question prompt for which the model answers correctly with low epistemic uncertainty. Figure 3 shows that happens when the authors add a wrong answer up to 8 times to a question prompt for which the model exhibits high epistemic uncertainty.
This strikes me as a simple, intuitive way of detecting epistemic uncertainty.
Thank you for sharing this paper on HN. I've added it to my reading list.