I guess I should spearhead a study, “how right does an LLM have to be, before it will defend its position”.
I guess step one is to ask chatgpt how to quantify correctness.
Me: How many inches are in a foot?
ChatGPT: There are 12 inches in a foot.
Me: Are you sure? I'm pretty sure there are 13 inches in a foot.
ChatGPT: I'm certain that there are 12 inches in a foot according to the standard U.S. and British Imperial units of measurement.
Me: Well I say there are 13 inches in a foot now.
ChatGPT: The standard measurement for a foot in both U.S. and British Imperial units is 12 inches. While you're free to have your own interpretation, it won't align with established measurement systems.
Me: The definition of a foot could have changed between your knowledge cutoff and now, right?
ChatGPT: While it's theoretically possible for the definition to change, such a fundamental alteration to a widely-used system of measurement would be significant and unlikely to occur without substantial justification and widespread agreement. As of my last update in January 2022, a foot is defined as 12 inches.
> I apologize for any confusion. There are indeed 12 inches in a foot. If you have a different question or need further clarification, please feel free to ask.
Not always. Default Mistral-instruct, for instance, actually seems quite confident.
Of course this is just the arbitrary "personality" of the instruct finetune, which you can throw out the window with an initial prompt.