Hacker News Reader

Comment by AnotherGoodName | Hacker News Reader

ChatGPT in particular will give an incorrect (but unique!) answer every time. At the risk of losing a great example of AI hallucination, it's Autosketch

Not shown fully but https://www.youtube.com/watch?v=kBCrVwnV5DU&t=39s note the game in the file menu.

View on HN

Wow, that is quite obscure. Even with the name I can't find any references to it on Google. I'm not surprised that the LLMs don't know about it.

You can always make stuff up to trigger AI hallucinations, like 'which 1990s TV show had a talking hairbrush character?'. There's no difference between 'not in the training set' and 'not real'.

Edit: Wait, no, there actually was a 1990s TV show with a talking hairbrush character: https://en.wikipedia.org/wiki/The_Toothbrush_Family

This is hard.

> There's no difference between 'not in the training set' and 'not real'.

I know what you meant but this is the whole point of this conversation. There is a huge difference between "no results found" and a confident "that never happened", and if new LLMs are trained on old ones saying the latter then they will be trained on bad data.

>> You can always make stuff up to trigger AI hallucinations

Not being able to find an answer to a made up question would be OK, it's ALWAYS finding an answer with complete confidence that is a major problem.

I imagine asking for anything obscure where there's plenty of noise can cause hallucinations. What Google search provides the answer? If the answer isn't in the training data, what do you expect? Do you ask people obscure questions, and do you then feel better than them when they guess wrong?

I just tried:

  What MS-DOS program contains an easter-egg of an Amiga game?

And got some lovely answers from ChatGPT and Gemini.

Aside I personally would associate "productivity program" with productivity suite (like MS Works) so I would have trouble googling an answer (I started as a kid on Apple ][ and have worked with computers ever since so my ignorance is not age or skill related).

The good option would be for the LLM to say it doesn't know. It's the making up answers that's the problem.

interesting. gemini 2.5 pro considered that it might be "AutoCAD" but decided it was not:

"A specific user recollection of playing "Connect Four" within a version of AutoCAD for DOS was investigated. While this suggests the possibility of such a game existing within that specific computer-aided design (CAD) program, no widespread documentation or confirmation of this feature as a standard component of AutoCAD could be found. It is plausible that this was a result of a third-party add-on, a custom AutoLISP routine (a scripting language used in AutoCAD), or a misremembered detail."

I wouldn't worry about losing examples. These things are Mandela Effect personified. Anything that is generally unknown and somewhat counterintuitive will be Hallucination Central. It can't NOT be.

In what world is that 'productivity software'?

Sure, it helps you do a job more productively, but that's roughly all non-entertainment software. And sure, it helps a user create documents, but, again, most non-entertainment software.

Even in the age of AI, GIGO holds.

Debatable but regardless you could reformulate the question however you want and still won't get anything other than hallucinations fwiw since there's no references to this on the internet. You need to load up autosketch 2.0 in a dos emulator and see it for yourself.

Amusingly i get an authoritative but incorrect "It's autocad!" if i narrow down the question to program commonly used by engineers that had connect four built in.

"Productivity software" typically refers to any software used for work rather than entertainment. It doesn't mean software such as a todo list or organizer. Look up any laptop review and you'll find they segment benchmarks between gaming and "productivity". Just because you personally haven't heard of it doesn't mean it's not a widely used term.

https://en.m.wikipedia.org/wiki/Productivity_software

> Productivity software (also called personal productivity software or office productivity software) is application software used for producing information (such as documents, presentations, worksheets, databases, charts, graphs, digital paintings, electronic music and digital video). Its names arose from it increasing productivity

Reply on news.ycombinator.com