So....like a person?
And in an honest interaction, they will tell you.
ChatGPT etc. does not. So basically it acts like a pathological liar (who happen to be right as long as it has been trained on something that comes close enough).
Scientist? STEM workers? Maybe. People who value truth / consistent world model for its own sake? Sure.
Normies? Not so much. It's not that they can't - I think they never learned to pay enough attention. What I mean is, most people tend to say things in confidence, and maybe even believe them, regardless of how they acquired the information. They don't seem to process the distinction between "I read it in a book", "A colleague told me that their colleague heard on the radio that...", etc. They don't even track provenance of the information, which is a critical skill you need to not end up confidently making things up.
It's a learnable skill, and I think it's even quite easy to pick it up from osmosis - but someone has to make the person feel that it's important.
> So basically it acts like a pathological liar
I think it's more of a bullshitter than a liar, in the sense that it doesn't care about truth value of what it says.
Lying involve knowing the truth, or at least knowing that the thing you're saying ain't it. Making a mistake involves you thinking you're saying the truth, but actually being wrong about it. Bullshitting is just saying whatever helps you achieve your goal; the truth value of what you say doesn't even enter the picture.