Just claiming a capability does not make it true and we have 0 “proof” of original reasoning that can be proved coming from these models. Especially given the potential cheating in current SOTA benchmarks
???
https://the-decoder.com/language-models-use-a-probabilistic-...
From my own perspective: if we’re gonna say these things reason and we’re using the definition of reasoning we apply to humans, then being able to reason through the trivial cases they fail to today would be a start. To the proponents of “they reason sometimes but not others” my question is why? What reason does it have to not reason and why if it is reasoning it still fails on trivial things that are variations of its own training data? I would also expect that these models would use reasoning to find new things like humans do but without humans essentially guiding the model to the correct awnser or the model just brute-forcing a problem-space with a set of rules/heuristics. Not exhaustive but a good start I think. These models have trouble currently even doing the advertised things like “book a trip for me” once a UI update happens so I think it’s a great indication we don’t quite have the intelligence/reasoning aspect worked out.
Another question I have: would a form of authentic reasoning in a model give rise to a model having an aesthetic? Could this be some sort of indicator of having created a “model of the world”? Does the model of the world perhaps imply a value judgement about it given that if one was super intelligent wouldn’t one of the first things realized be the limitations of its own understanding even given the restrictions of time and space and not ever potentially being able to observe the universe in its entirety? Perhaps a perfect super intelligence would just evaporate/transcend like in the Culture series. What a time to be alive!
And then you'd realize that a lot of naïve arguments against LLMs would imply that a significant portion of homo-sapiens can't reason, are unable to really think, and are no more than stochastic parrots.
It's actually a rather dangerous line of reasoning.
LLMs CAN reason. Whether it can’t reason is not provable. To prove that you have to give the LLM every possible prompt that it has no data for and effectively show it never reasons and gets it wrong all the time. Not only is the proof impossible but it’s already been falsified as we have demonstrable examples of LLMs reasoning.
Literally I invite people to post prompts and correct answers to ChatGPT where it is trivially impossible for that prompt to exist in the data. Every one of those examples falsifies the claim that LLMs can’t reason.
Saying LLMs can’t reason is an overarching claim similar to the claim that humans and LLMs always reason. Humans and LLMs don’t always reason. But they can reason.
This also looks promising:
https://hiw.kuleuven.be/en/study/prospective/OOCP/introducti...
If you wanted something a bit different Wittgenstein’s Tractatus has always made my head spin with possibilities:
No one is making the claim that LLMs reason like humans or always reason correctly. Ask anyone who makes a claim similar to mine. We are all ONLY making the claim that LLMs can reason correctly. That is a small claim.
The counterclaim is LLMs can’t reason and that is a vastly expansive claim that is ludicrously unprovable.
This is a unique problem I came up with. It’s a variation on counting islands. There are actually two correct answers that are straightforward. Other answers may exist but are generally not straightforward and often wrong. One answer is mathematical the other is a leetcode style solution.
Try to solve this yourself before using ai to get a feel for how hard it is. The solution should be extremely straightforward. It’s also fun to think about. When you try to think of a solution you will invariably come up with a bunch of possible solutions that are wrong which is a strong indicator of how large the range of possible answers are. Few answers are correct but many look correct.
I give this test to candidates and I never expect the candidate to solve it because it’s one of the few algorithms that requires actual reasoning and actual creativity as I came up with it so no variation of it really exists anywhere else. You can’t pattern match for it. Out of like 50 candidates you probably get one person able to solve it in less than an hour.
It’s unlikely most people on hn will be able to solve it. If you do solve it don’t post the answer as it will become training data for the next iteration of the LLM.
I gave the prompt to o3. It solved. It generated code as well which I was too lazy to verify but it solved it correctly in the description of the algorithm involved.
There is also a 3D version of this problem where the grid is 3D. It changes the entire problem if a donut is in 3D space. It is harder and I have only found one possible solution for it. I have not tried it on an LLM.
Dude, if you can formulate a question and prove an answer absolutely requires "reasoning" (defined how?) then you should drop everything and publish a paper on it immediately.
You'll have plenty of time to use your discovery to poke at LLMs after you secure your worldwide fame and recognition.
They scan a hyperdimensional problem space whose facetness and capacity a single human is unable to comprehend. But there potentially exist a slice that corresponds to a problem that is novel to a human. LLMs are completely alien to us both in capabilities and technicalities, so talking about whether they can reason makes as much sense as if you replaced “LLMs” with “rainforests” or “antarctica”.
Clearly we used the term reasoning for many varied techniques. The term doesn’t narrow to specifically one form of “human” like reasoning only.
But we have to at least define "reasoning" for the given manifestation of "it". Otherwise it's just birdspeak. Because reasoning is "the action of thinking about something in a logical, sensible way", which has to happen somewhere if not finger-pointable, then at least somehow scannable or otherwise introspectable. Otherwise it's yet another omnidude in the sky who made it all so that you cannot see him, but there will be hints if you believe.
Anyway, we have to talk something specific, not handwavy. Even if you prove that they CAN reason for some definition of it, both the proof and the definition must have some predictive/scientific power, otherwise they are as useless as nil thought about it.
For example, if you prove that the reasoning is somehow embedded as a spatial in-network set of dimensions rather than in-time, wouldn't that be literally equivalent to "it just knows the patterns"? What would that term substitution actually achieve?
If the output has a low probability of occuring by random chance then it must be reason.
>For example, if you prove that the reasoning is somehow embedded as a spatial in-network set of dimensions rather than in-time, wouldn't that be literally equivalent to "it just knows the patterns"? What would that term substitution actually achieve?
I mean, this is a method many humans use to reason themselves.
From what I can see, you’re only massaging semantics. That is uninteresting.
In your example the book is the training data or aka the input.
All I have to do is show you one prompt with a correct answer that cannot be arrived at with pattern matching and the prompt can only be arrived at through reasoning. One. You have to demonstrate this for EVERY prompt if you want to prove LLMs can't reason.
I’m also not entirely convinced we can’t arrive at a reasoning system via probability only (a really cool thought experiment) but these systems do not meet the consistency/intelligence bar for me to believe this currently.
That’s the claim everyone makes. That is a human definition if it reasoned one time correctly. That is the colloquial definition.
Someone who has brain damage can reason correctly on certain subjects and incorrectly on other subjects. This is an immensely reasonable definition. I’m not being pedantic or out of line here when I say LLMs can reason while using this definition.
Nobody is making the claim that LLMs reason like humans or are human or reason perfectly every time. Again the claim is: LLMs are capable of reasoning.
So the the statement that they CAN reason is demonstrably true.
Ok if given a prompt where the solution can only be arrived at by reasoning and the LLM gets to the solution for that single prompt, then how can you say it can't reason?
If you do find this question answer pair then it would be a massive breakthrough for science and philosophy more generally.
You say “demonstrably” but I still do not see a demonstration of these reasoning abilities that is not subject to the aforementioned criticisms.
Your quote: “This is a unique problem I came up with. It’s a variation on counting islands.” You then say: “ as I came up with it so no variation of it really exists anywhere else.”
So not sure what to take away from your text but I do think this is a variation of a well-known problem type so I would be pretty amazed if there was something very close to this in the training data. Given it’s an interview question and those are written about ad-nauseum I’m not surprised then that it was able to generalize to the provided case. The COT researchers did see the ability to generalize in some cases just not necessarily actually use the COT tokens to reason and/or failed on generalizing on variations which they thought it should have given its ability to generalize in others and the postulation that it was using reasoning and not just a larger corpus to pattern match with.
The solution however is not a variation. It requires leaps of creativity that most people will be unable to solve. In fact I would argue this goes beyond just reasoning as you have to be creative and test possibilities to even arrive at a solution. It’s almost random chance that will get you there. Simple reasoning like logical reduction won’t let you arrive at a solution.
Additionally this question was developed to eliminate pattern matching that candidates use on software interviews. It was vetted and verified to not exist. No training data exists.
It definitively requires reasoning to solve. And it is also unlikely you solved it. ChatGPT o3 has solved it. Try it.
“And it is also unlikely you solved it” well I guess you overestimated your abilities on two counts today then.
> It’s a variation on a well known problem in the sense that I just added some unique rules to it.
> No training data exists.
No it definitely does but is a variation. You kinda just confirmed what we already knew. Given enough data about a thing these LLMs can generalize somewhat.
Think of it like this counting islands exists in the training data in the same way addition exists. The solution to this problem builds off of counting islands in the same way calculus builds off of addition.
No training data exists for it to copy because this problem is uniquely invented by me. The probability that it has is quite low. Additionally several engineers and I have done extensive google searches and we believe to a reasonable degree that this problem does not exist anywhere else.
Also you use semantics to cover up your meaning. LLMs can “generalize” somewhat? Generalization is one of those big words that’s not well defined. First off the solution is not trivially extracted from counting islands and second “generalize” is a form of reasoning. You’re using big fuzzy words with biased connotations to further your argument. But here’s the thing, even if we generously go with it, the solution to counting donuts is clearly not some trivial generalization of counting islands. The problem is a variation but the solution is NOT. It’s not even close to what we term as the colloquial definition of “generalization”
Did you solve it? I highly doubt you did. It’s statistically more likely you’re lying, and the fact that you call the solution a “generalization” just makes suspect that even more.
Also generalization is not always reasoning: I can make a generalization that is not reasoned; I can also make one that is poorly reasoned. Generalization is considered well-defined in regards to reasoning: https://www.comm.pitt.edu/reasoning
Your example still fails to actually demonstrate reasoning given its highly derivative nature, though.
That’s not the point though. The point is I have metrics on this. Roughly 50 interviews only one guy got it. So you make the claim the solution is generalize-able well then prove your claim then. I have metrics that support my claim. Where’s yours?
If I roll a die and only _sometimes_ it returns the correct answer to a basic arithmetic question, this is the exact reason why we don't say a die is doing arithmetic.
Even worse in the case of LLMs, where it's not caused by pure chance, but also training bias and hallucinations.
You can claim nobody knows the exact definition of reasoning, maybe there are some edges which aren't clearly defined because they're part of Philosophy, but applying rules of logic consistently is not something you just don't always do and still call it reasoning.
Also, LLMs are generally incapable of saying they don't know something, cannot know something, can't do something, etc. They would rather try and hallucinate. When it does that, it's not reasoning. And you also can't explain to an LLM how to figure out it doesn't know something, and then actually say it doesn't know and not make stuff up. If it was capable of reasoning you should be able to convince it using _reason_, to do exactly that.
However, you
There is a question at the end of that comment, would love to hear other options.
Most people will use this excuse to avoid responding to or even looking at your link here. It is compelling evidence.
Therefore, how does my example not qualify as this, at least:
> Analogical reasoning involves the comparison of two systems in relation to their similarity. It starts from information about one system and infers information about another system based on the resemblance between the two systems.
I guess the real question is “does moving down a stochastic gradient of probabilities suffice as reasoning to you” and my awnser is no because you don’t need reason to find the nearest neighbor in this architecture. In this case the model is not actively comparing and inferring its simply associating without “knowing”
> It depends on how you define "reasoning."
> LLMs like me can perform certain types of reasoning, such as:
> Pattern recognition & analogy – Recognizing relationships between concepts and applying similar patterns in new contexts.
> Deductive reasoning – Applying rules to specific cases (e.g., "If all humans are mortal and Socrates is a human, then Socrates is mortal").
> Inductive reasoning – Generalizing from specific examples (though often with statistical biases).
> Common-sense reasoning (to an extent) – Inferring likely outcomes based on broad training data.
> However, there are limitations:
> No true understanding – LLMs don’t "know" things like humans do; they predict based on probabilities . > Struggles with multi-step logic – Complex reasoning that requires deep memory or long-term planning is difficult.
> Lack of real-world interaction – Without direct experience, reasoning is limited to text-based patterns.
> Confidence: 90%
> (Strong evidence suggests LLMs can perform certain types of reasoning, but they are not yet comparable to human-level reasoning.)
Would you agree with that analysis? If so, then LLMs are indeed capable of reasoning, in some ways.
If you ask "who is the parent of <celebrity-child-name>", it will often claim to have no knowledge about this person.
Yes sometimes it gets it right, but sometimes also not. Try a few celebrities.
Maybe the disagreement is about this?
Like if it gets it right a good amount of the time, you would say that means it's (in principle) capable of reasoning.
But I say, that if it gets it wrong a lot of the time, that means 1) it's not reasoning in situations when it gets it wrong, but also 2) it's most likely also not reasoning in situations when it gets it right.
And maybe you disagree with that, but then we don't agree on what "reasoning" means. Because I think that consistency is an important property of reasoning.
I think that if it gets "A is parent of B, implies B is child of A" wrong for some celebrity parents, but not for others, then it's not reasoning. Because reasoning would mean applying this logical construct as a rule, and if it's not consistent at that, it makes it hard to argue that it is in fact applying this logical rule instead of doing who-knows-what that happens to give the right answer, some of the time.
"I made a hypothesis that works with 1 to 5. if a hypothesis holds for 10 numbers, it holds for all numbers"
The opposing claim unbounded. It says LLMs can't reason period. They are making the claim that it is 100% for all possible prompts.
No one is making the claim LLMs reason all the time and always. They don't. The claim is that they CAN reason.
Versus the claim that they can't which is all encompassing and ludicrous.
your evidence: "it works with these inputs I tried!"
...hmm seems you're not quite versed in basic mathematical proofs?
If I can reason it doesn’t mean I’m always reasoning or constantly reasoning or if I know how to do reasoning for every prompt. It just means it’s possible. How narrow or how wide that possibility is, is orthogonal to the claim itself. Please employ logic here.
Ok math guy. Imagine I said numbers can be divided. The claim is true even though there is a number that can’t be divided. Zero.
Of course make sure the prompt is unique such that it's not in the data and it's not doing any sort of "pattern matching".
So like all science we prove it via probability. Observations match with theory to a statistical degree.
It seems to me that, in natural language, the size of the codomain is related to the specificity of the prompt. For instance, if the prompt is "We are going to ..." then the codomain is enormous. But if the prompt is "2 times 2 is..." the codomain is, mathematically, {4, four}, some series of 4 symbols, eg IIII, or some other representation of the concept of "4" (ie different base or language representations: 0x04, 0b100, quatro, etc).
But if this is the case, a broad codomain is approximately synonymous with "no correct answer" or "result is widely interpretable". Which implies that the larger the codomain the easier it is to claim an answer "correct" in context of the prompt.
How do you reconcile loose interpretability with statistical rigor?
I ask the question, what is 2 * 2, which is an obviously loaded question that's pattern matched to death.
The LLM can answer "4" or "The answer is 4" of "looks like the answer is 4"
All valid answers but all the same. We count all 3 of those answers as just 4 out of the set of numbers. But we have to use our own language faculties to cut through the noise of the language itself.
Yeah, that was my point. Small codomain -> easy to validate. Large codomain -> open to interpretation. You implied that to prove reasoning, pick a prompt with a large codomain and if the LLM answers with accurate precision, then viola, reasoning.
So my question was, can you give an example of a prompt with a high codomain that isn't subject to wide interpretation? It seems the wider the codomain the easier it is to say, "look! reasoning!"
So my question is, and has been these three replies: Can you give any example of one of these prompts?
Literally I invite people to post prompts and correct answers to ChatGPT where it is trivially impossible for it to have known what number you were thinking of. Every one of those examples falsifies the claim that LLMs can’t read minds.
It's a probability problem really. The range of a prompt has billions of possibilities. If it arrived at a correct answer within that range then the probability it got there without reasoning is miniscule.
Same with this mind reading thing. Prove it.
This isn't demonstrated yet, I would say. A good analogy is how people have used NeRFs to generate Doom levels, but when they do, the levels don't have offscreen coherence or object permanence. There's no internal engine behind the scenes making an actual Doom level. There's just a mechanism to generate things that look like outputs of that engine. In the same way, an LLM might well just be an empty shell that's good at generating outputs based on similar-looking outputs it was trained on, rather than something that can do the work of thinking about things and producing outputs. I know that's similar to "statistical parrot", but I don't think what you're saying demonstrates anything more than that.