Comment by godelski | Hacker News Reader

Comment by godelski | Hacker News Reader

Parent Full thread

There were also several hallucinations during the announcement. (I also see hallucinations every time I use Claude and GPT, which is several times a week. Paid and free tiers)

So not seeing them means either lying or incompetent. I always try to attribute to stupidity rather than malice (Hanlon's razor).

The big problem of LLMs is that they optimize human preference. This means they optimize for hidden errors.

Personally I'm really cautious about using tools that have stealthy failure modes. They just lead to many problems and lots of wasted hours debugging, even when failure rates are low. It just causes everything to slow down for me as I'm double checking everything and need to be much more meticulous if I know it's hard to see. It's like having a line of Python indented with an inconsistent white space character. Impossible to see. But what if you didn't have the interpreter telling you which line you failed on or being able to search or highlight these different characters. At least in this case you'd know there's an error. It's hard enough dealing with human generated invisible errors, but this just seems to perpetuate the LGTM crowd

What were the hallucinations during the announcement?

My incompetence here was that I was careless with my use of the term "hallucination" here. I assumed everyone else shared my exact definition - that a hallucination is when a model confidently states a fact that is entirely unconnected from reality, which is a different issue from a mistake ("how many Bs in blueberry" etc).

It's clear that MANY people do not share my definition! I deeply regret including that note in my post.

I subscribe to the same definition as you. I've actually never heard someone referring to the mistakes as hallucinating until now, but I can see how it's a bit of a grey area.

I'm actually curious about how you both come to those definitions of hallucinations. It gets very difficult to distinguish these things when you dig into them. Simon dropped this paper[0] in another thread[1] and while they provide a formal mathematical definition I don't think this makes it clear (I mean it is one person (who doesn't have a long publication record, PhD, or work at a university), but following their definition is still a bit muddy. They say the truth has to be in the training but don't clarify if they mean in training distribution or literal training example.

To make a clear example, is the fact that when prompting GPT-5 with "Solve 5.x = x + 5.11" it answers "-0.21" (making the same mistake as when it GPT-4 says 5.11 > 5.9). Is that example specifically in the training data? Who knows! But are those types of problems in the training data? Absolutely! So is this a mistake or a hallucination? Should we really be using an answer that requires knowing the exact details of the training data? That would be fruitless and allow any hallucination to be claimed as a mistake. But in distribution? Well that works because we can know the types of problems trained on. It is also much more useful given that the reason we build these machines is for generalization.

But even without that ambiguity I think it still gets difficult to differentiate a mistake from a hallucination. So it is unclear to me (and presumably others) what the precise distinction is to you and Simon.

[0] https://arxiv.org/abs/2508.01781

[1] https://news.ycombinator.com/item?id=44831621

Yeah, I agree with you with you dig down into it.

But I tend to instinctually (as a mere human) think of a "hallucination" as something more akin to a statement that feels like it could be true, and can't be verified by using only the surrounding context -- like when a human mis-remembers a fact on something they recently read, or extrapolates reasonably, but incorrectly. Example: GPT-5 just told me a few moments ago that webpack's "enhanced-resolve has an internal helper called getPackage.json". Webpack likely does contain logic that finds the package root, but it does not contain a file with this name, and never has. A reasonable person couldn't say with absolutely certainty that enhanced-resolve doesn't contain a file with that name.

I think a "mistake" is classified as more of an error in computation, where all of the facts required to come up with a solution are present in the context of the conversation (simple arithmetic problems, "how many 'r's in strawberry", etc.), but it just does it wrong. I think of mistakes as something with one and only one valid answer. A person with the ability to make the computation themselves can recognize the mistake without further research.

So hallucinations are more about conversational errors, and mistakes are more about computational errors, I guess?

But again, I agree, it gets very difficult to distinguish these things when you dig into them.

The reason it gets very difficult to distinguish between the two is that there is nothing to distinguish between the two other than subjective human judgement.

When you try to be objective about it, it's some input, going through the same model, producing an invalid statement. They are not different in no way, shape or form, from a technical level. They can't be tackled separately because they are the same thing.

So the problem of distinguishing between these two "classes of errors" reduces to the problem of "convincing everyone else to agree with me". Which, as we all know, is next to impossible.

I can't pinpoint exactly where I learned my definition of hallucination - it's been a couple of years I think - but it's been constantly reinforced by conversations I've had since then, to the point that I was genuinely surprise in the past 24 hours to learn that a sizable number of people categorize any mistake by a model as a hallucination.

See also my Twitter vibe-check poll: https://twitter.com/simonw/status/1953565571934826787

Actually... here's everything I've written about hallucination on my blog: https://simonwillison.net/tags/hallucinations/

It looks like my first post that tried to define hallucination was this one from March 2023: https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces...

Where I outsourced the definition by linking to this Wikipedia page: https://en.m.wikipedia.org/wiki/Hallucination_(artificial_in...

You must have missed the ridiculous graphs, or the Bernoulli error, while these corpo techno fascists were buying your dinner.

https://news.ycombinator.com/item?id=44830684

https://news.ycombinator.com/item?id=44829144

The graphs were nothing to do with model hallucination, that was a crap design decision by a human being.

The Bernoulli error was a case of a model spitting out widely believed existing misinformation. That doesn't fit my mental model of a "hallucination" either - I see a hallucination as a model inventing something that's not true with no basis in information it has been exposed to before.

Here's an example of a hallucination in a demo: that time when Google Bard claimed that the James Webb Space Telescope was first to take pictures of planet outside Earth’s solar system. That's plain not true, and I doubt they had trained on text that said it was true.

I don't care what you call each failure mode. I want something that doesn't fail to give correct outputs 1/3 to 1/2 the time.

Forget AI/AGI/ASI, forget "hallucinations", forget "scaling laws". Just give me software that does what it says it does, like writing code to spec.

Along those lines, I also want something that will correct me if I am wrong. The same way a human would or even the same way Google does because typing in something wrong usually has enough terms to get me to the right thing, though usually takes a bit longer. I definitely don't want something that will just go along with me when I'm wrong and reinforce a misconception. When I'm wrong I want to be corrected sooner than later, that's the only way to be less wrong.

You might find this updated section of the Claude system prompt interesting: https://gist.github.com/simonw/49dc0123209932fdda70e0425ab01...

> Claude critically evaluates any theories, claims, and ideas presented to it rather than automatically agreeing or praising them. When presented with dubious, incorrect, ambiguous, or unverifiable theories, claims, or ideas, Claude respectfully points out flaws, factual errors, lack of evidence, or lack of clarity rather than validating them. Claude prioritizes truthfulness and accuracy over agreeability, and does not tell people that incorrect theories are true just to be polite.

No idea how well that actually works though!

Considering your other comment, you may not consider it a hallucination but the fact about the airfoil was wrong. I'm sure there was information in the training that had the same mistake because that mistake exists in textbooks BUT I'm also confident that the correct fact is in the training as you can get GPT to reproduce the correct fact. The hallucination most likely happened because the prompt primes the model for the incorrect answer by asking about Bernoulli.

But following that, the "airfoil" it generates for the simulation is symmetric. That is both inconsistent with its answers and inconsistent with reality, so I think that one is more clear.

Similarly, in the coding demo the French guy even says that the snake doesn't look like a mouse haha.

GPT-5 did the below today when I asked it to proofread an email. Would you consider it a hallucination? I'm asking genuinely because I'm a fan of your thoughts on tech and am curious where you draw the line.

My wording: "Would you have time to talk the week of the 25th?"

ChatGPT wording (elipses mine): "Could we schedule ~25 minutes the week of Aug 25 [...]? I’m free Tue 8/26 10:00–12:00 ET or Thu 8/28 2:00–4:00 ET, but happy to work around your calendar."

I am not, in fact, free during those times. I have seen this exact kind of error multiple times.

You can just have a different use case that surfaces hallucinations than someone, they don’t have to by evil.

Reply on news.ycombinator.com