Hacker News Reader

Comment by nextaccountic | Hacker News Reader

The tool may malfunction. Indeed there are theorems that say the tool must malfunction at least sometimes.

If you don't understand the subject and relies on the understanding of the tool, how will you have a frigging chance of catching an error?

Anyway this reminds me that scene in Matrix 2, where the mayor (of sorts, don't remember his exact title) of Zion tells Neo about the machines that keeps everyone alive, but that there is nobody left that understands how they work or could repair them if they broke.

View on HN

>Indeed there are theorems that say the tool must malfunction at least sometimes.

Gödel's incompleteness theorem only states that there are true statements that can never be proved to be true. That limits what tools can do, but it's not a "malfunction". The exact same limitation applies to humans. A genuine malfunction would be if the tool "proved" a false statement to be true. That actually has happened before because of bugs in proof checkers, but software bugs can be found and fixed, and proof checkers are simple enough that it's possible to write many different implementations. It's extraordinarily unlikely that the same malfunction occurs in all of them.

I'm not talking about incompleteness though. I'm broadly talking about limitations of LLMs. For example, this paper from this year https://www.nature.com/articles/s41586-026-10549-w says that the current training regime incentivizes hallucinations and can't make them go to zero. Well in the pre-training phase at least (but I guess you could make a similar argument for post-training with RL)

It doesn't matter. The proof verifier is traditional deterministic software with 0% hallucination rate, so any hallucinated proofs are easily removed.

I'm not talking about proofs. I was answering to this

> Why not? If mathematicians in the field cannot understand the papers, but tools at our disposal can, why not use them? What is "unhealthy" about that? Just seems like it's the reality of the world now.

Which talked about using LLMs to understand math, not to prove theorems

edit: to be clearer, on my areas of expertise, I regularly spot LLMs making false or misleading statements, which usually comes accompanied with a lot of truthful ones. If I push back, they often backtrack and admit they were wrong.

On areas I do not have expertise, I can't be a good judge. And this was my initial point: if you don't understand the subject, how will you catch errors?

So I don't think you can or should use to replace understanding the subject yourself. It's unhealthy to offload your thinking to a tool because it makes you kind of useless.

Reply on news.ycombinator.com