Not quite ... LLMs are not HAL (unfortunately). They produce something that is associated with the same input, something that should look like an acceptable answer. A correct answer will be acceptable, and so will any answer that has been associated with similar input. And so will anything that fools some of the people, some of the time ;)
The unpredictability is a huge problem. Take the geoguess example - it has come up with a collection of "facts" about Paramaribo. These may or may-not be correct. But some are not shown in the image. Very likely the "answer" is derived from completely different factors, and the "explanation" in spurious (perhaps an explanation of how other people made a similar guess!)
The questioner has no way of telling if the "explanation" was actually the logic used. (It wasn't!) And when genuine experts follow the trail of token activation, the answer and the explanation are quite independent.
This is very important and often overlooked idea. And it is 100% correct, even admitted by Anthropic themselves. When user asks LLM to explain how it arrived to a particular answer, it produces steps which are completely unrelated to the actual mechanism inside LLM programming. It will be yet another generated output, based on the training data.
Huh? If I need to sort the list of integer number of 3,1,2 in ascending order the only correct answer is 1,2,3. And there are multiple programming and mathematical questions with only one correct answer.
If you want to say "some programming and mathematical questions have several correct answers" that might hold.
1, 2, 3
1,2,3
[1,2,3]
1 2 3
etc.
"1 2 3" is another
"After sorting, we get `1, 2, 3`" yet another
etc.
At least, that's how I understood GP's comment.
- In Math, there's often more than one logically distinct way of proving a theorem, and definitely many ways of writing the same proof, though the second applies more to handwritten/text proofs than say a proof in Lean.
- In programming, there's often multiple algorithms to solve a problem correctly (in the mathematical sense, optimality aside), and for the same algorithm there are many ways to implement it.
LLMs however are not performing any logical pass on their output, so they have no way of constraining correctness while being able to produce different outputs for the same question.
Yes, I thought as well of your interpretation, but then I read the text again, and it really does not say that, so I choose to answer to the text...
Neither do we.
> They will give a solution that seems correct under their heuristic reasoning, but they arrived at that result in a non-logical way.
As do we, and so you can correctly reframe the issue as "there's a gap between the quality of AI heuristics and the quality of human heuristics". That the gap is still shrinking though.
And no, just because you can imagine a human stupid enough to make the same mistake, doesn't mean that LLMs are somehow human in their flaws.
> the gap is still shrinking though
I can tell this human is fond of extrapolation. If the gap is getting smaller, surely soon it will be zero, right?
I don't believe anyone is suggesting that LLMs flaws are perfectly 1:1 aligned with human flaws, just that both do have flaws.
> If the gap is getting smaller, surely soon it will be zero, right?
The gap between y=x^2 and y=-x^2-1 gets closer for a bit, fails to ever become zero, then gets bigger.
The difference between any given human (or even all humans) and AI will never be zero: Some future AI that can only do what one or all of us can do, can be trivially glued to any of that other stuff where AI can already do better, like chess and go (and stuff simple computers can do better, like arithmetic).
Ditto for your mischaracterizations of LLMs.
> There are still so many obvious errors (noticeable by just using Claude or ChatGPT to do some non-trivial task) that the average human would simply not make.
Firstly, so what? LLMs also do things no human could do.
Secondly, they've learned from unimodal data sets which don't have the rich semantic content that humans are exposed to (not to mention born with due to evolution). Questions that cross modal boundaries are expected to be wrong.
> If the gap is getting smaller, surely soon it will be zero, right?
Quantify "soon".
citation needed
Human errors in judgement can also be discovered, explained, and reverted.