Hacker News Reader

Comment by Legend2440 | Hacker News Reader

ChatGPT 4o waffles a little bit and suggests the Microsoft Entertainment pack (which is not productivity software or MS-DOS), but says at the end:

>If you're strictly talking about MS-DOS-only productivity software, there’s no widely known MS-DOS productivity app that officially had a built-in Connect Four game. Most MS-DOS apps were quite lean and focused, and games were generally separate.

I suspect this is the correct answer, because I can't find any MS-DOS Connect Four easter eggs by googling. I might be missing something obscure, but generally if I can't find it by Googling I wouldn't expect an LLM to know it.

View on HN

ChatGPT in particular will give an incorrect (but unique!) answer every time. At the risk of losing a great example of AI hallucination, it's Autosketch

Not shown fully but https://www.youtube.com/watch?v=kBCrVwnV5DU&t=39s note the game in the file menu.

Wow, that is quite obscure. Even with the name I can't find any references to it on Google. I'm not surprised that the LLMs don't know about it.

You can always make stuff up to trigger AI hallucinations, like 'which 1990s TV show had a talking hairbrush character?'. There's no difference between 'not in the training set' and 'not real'.

Edit: Wait, no, there actually was a 1990s TV show with a talking hairbrush character: https://en.wikipedia.org/wiki/The_Toothbrush_Family

This is hard.

> There's no difference between 'not in the training set' and 'not real'.

I know what you meant but this is the whole point of this conversation. There is a huge difference between "no results found" and a confident "that never happened", and if new LLMs are trained on old ones saying the latter then they will be trained on bad data.

>> You can always make stuff up to trigger AI hallucinations

Not being able to find an answer to a made up question would be OK, it's ALWAYS finding an answer with complete confidence that is a major problem.

interesting. gemini 2.5 pro considered that it might be "AutoCAD" but decided it was not:

"A specific user recollection of playing "Connect Four" within a version of AutoCAD for DOS was investigated. While this suggests the possibility of such a game existing within that specific computer-aided design (CAD) program, no widespread documentation or confirmation of this feature as a standard component of AutoCAD could be found. It is plausible that this was a result of a third-party add-on, a custom AutoLISP routine (a scripting language used in AutoCAD), or a misremembered detail."

In what world is that 'productivity software'?

Sure, it helps you do a job more productively, but that's roughly all non-entertainment software. And sure, it helps a user create documents, but, again, most non-entertainment software.

Even in the age of AI, GIGO holds.

Debatable but regardless you could reformulate the question however you want and still won't get anything other than hallucinations fwiw since there's no references to this on the internet. You need to load up autosketch 2.0 in a dos emulator and see it for yourself.

Amusingly i get an authoritative but incorrect "It's autocad!" if i narrow down the question to program commonly used by engineers that had connect four built in.

"Productivity software" typically refers to any software used for work rather than entertainment. It doesn't mean software such as a todo list or organizer. Look up any laptop review and you'll find they segment benchmarks between gaming and "productivity". Just because you personally haven't heard of it doesn't mean it's not a widely used term.

https://en.m.wikipedia.org/wiki/Productivity_software

> Productivity software (also called personal productivity software or office productivity software) is application software used for producing information (such as documents, presentations, worksheets, databases, charts, graphs, digital paintings, electronic music and digital video). Its names arose from it increasing productivity

I imagine asking for anything obscure where there's plenty of noise can cause hallucinations. What Google search provides the answer? If the answer isn't in the training data, what do you expect? Do you ask people obscure questions, and do you then feel better than them when they guess wrong?

I just tried:

  What MS-DOS program contains an easter-egg of an Amiga game?

And got some lovely answers from ChatGPT and Gemini.

Aside I personally would associate "productivity program" with productivity suite (like MS Works) so I would have trouble googling an answer (I started as a kid on Apple ][ and have worked with computers ever since so my ignorance is not age or skill related).

The good option would be for the LLM to say it doesn't know. It's the making up answers that's the problem.

I wouldn't worry about losing examples. These things are Mandela Effect personified. Anything that is generally unknown and somewhat counterintuitive will be Hallucination Central. It can't NOT be.

> I might be missing something obscure, but generally if I can't find it by Googling I wouldn't expect an LLM to know it.

The Google index is already polluted by LLM output, albeit unevenly, depending on the subject. It's only going to spread to all subjects as content farms go down the long tail of profitability, eking profits; Googling won't help because you'll almost always find a result that's wrong, as will LLMs that resort to searching.

Don't get me started on Google's AI answers that assert wrong information and launders fanfic/reddit/forum and elevating all sources to the same level.

It gave me two answers (one was Borland sidekick) which I then asked "are you sure about that?" waffled and said actually neither of those it's IBM Handshaker to which I said "I don't think so, I think it's another productivity program" and it replied on further review it's not IBM Handshaker, there are no productivity programs that include Connect Four. No wonder CTO like this shit so much, it's the perfect bootlick.

If I can find something by Googling I wouldn’t need an LLM to know it.

Any current question to an LLM is just a textual interpretation of the search results though; the use the same source of truth (or lies in many cases)

Reply on news.ycombinator.com