Hacker News Reader

Comment by breezybottom | Hacker News Reader

LLMs hallucinate and pander. Calling something "classic" is a common LLMism

So, just to confirm: you’re saying that it’s hallucinating “classic”, because you don’t think it’s in the training data, even though it correctly is a classic test?

Please show me your personal reasoning chain how you reached this conclusion because I think you’re hallucinating that the models are hallucinating the word “classic” here.

You're making a lot of assumptions.

I think it’s quite a stretch to say that when a model correctly calls out something to be a classic problem, that it’s probably hallucinating.

That’s quite a big assumption from your end as well.

It's not a classic problem though. It's a very niche, novel benchmark that can't be more than 2-3 years old. That doesn't fit any definition of "classic"

It will be two years old on the 25th of October.

I think it's credible to call it a "classic" though, in a field that moves this fast. Name another comedy benchmark for LLMs with the same traction.

Turing test

right? "im having this niche problem i find no information on anywhere, any ideas?" and you get "ahh this is the classic <wrong, other niche problem>"

Or “of course, super niche thing you made up is a problem everyone’s had” - they used to do that but not sure if anymore.

There's 8 billion people on this planet, and you're probably failing to imagine something sufficiently niche to be not actually very common anyway.

not all 8B have computer access, and even less access the computer as a device they control. Our crowd is actually rather small.

I meant “made up” as in truly invented - this was years ago now, they’d use a boilerplate sometimes with “everyone’s dealt with <x>”, but see what you mean.

Okay, but then "years ago" in AI is a different era entirely.

Reply on news.ycombinator.com