People usually do act as if they know what they're saying is incorrect, especially if it's a case of lack of recollection rather than a conscious and strategic lie. ChatGPT defaults to offering a continuation with the correct format and plausible text strings, without regard to truth values it doesn't know because it's indifferent to them.
And even if human learning or human motivation did lead a to person to internalise nothing about, say, the titles of scientific papers but the general format and authors well enough to make up similar sounding ones on demand in future, they probably wouldn't be as good as it...
Basically everything it says is a lie, some of those lies happen to randomly be true.
Edit: This also might be helpful: https://learn.microsoft.com/en-us/azure/cognitive-services/q...
Yes and no, the goal wasn’t focused on just being a source of truth. They wanted it to be able to write fairy tales, poetry, and song lyrics so creative answers where rewarded even if they weren’t factually accurate.
There was an attempt to reduce harmful and deceitful responses, but the team acknowledge the tendency for “plausible-sounding but incorrect or nonsensical answers” calling them hallucinations. Which becomes common when prompts are outside of it’s training set.
So yes there is definitely a link between the internal “Correctness” score and “Truth” but they are measuring very different things.