How do we know ChatGPT isn't hallucinating parts of its answers? I think this is an interesting service but what are the guardrails so we can trust the answer?
Having spent quite a lot of professional time working through real-world problems with GPT4, I'll say it very rarely does something I'd call hallucination. It almost never invents crazy things out of whole cloth. Sometimes it makes wrong guesses about things.
You gotta keep your own head and be skeptical. Often you can tell if it's coming unhinged. If it makes mistakes and you notice, it'll definitely own them unlike many LLMs. But of course it's not perfect.
When you add parts of the content of the PDF to the prompt, which is how these tools work, then it is very likely to produce a completion consistent with the PDF.
I say “mainly” and “very likely”, but in my experience I have never had a hallucination when the prompt is augmented with factual data.
Hallucinations occur with regularity when the prompt does not contain what is requested to be completed.
I get that people have an objection to how LLMs invent things, but so do people. You should always verify things.