Of course LLM's can't reason. They pattern match answers to previously asked questions, and humans will read the text as a reasonable answer because we assign meaning to it, but there is simply no way an LLM could use a "mental model" to "reason" about a problem other than constructing sentences out of probable matches it's been trained on.
That they are not effective at some forms of reasoning does not entail they can't reason.
Even reasoning, fundamental as it is, comes from feedback. Feedback from our actions teaches us how to reason. Learning from feedback is more general than reasoning - AI agents can definitely learn this way too, if they have enough freedom to explore. But you can't do it with supervised training sets alone.
You need to put language models into agents in environments, give them goals and rewards. Then they can make their own training data and mistakes, build up their own experience. Can't teach an AI based on how people make mistakes, it needs to fix its own mistakes, but that means to deploy it in the wild, where there are consequences for errors.
If you remember, DeepMind first tried to train a Go model on human game play, but it was limited. Then they started from scratch, and learning from feedback alone they surpassed human level, even though the feedback was one single bit of information at the end of a whole self-play game. And it had no pre-training prior to learning Go, unlike human players.
Essentially LLMs just are oracles for the shadows on Plato's cave.
Most of our intelligence is encoded in the environment and language, it's a collective process, not an individual one. We're collectively, not individually, very smart.
TLDR: Internal LLM representations correspond to an understanding of the visual world. We've all seen the Othello example, which is too constrained a world to mean much, but even more interesting is that LLMs can caption tokenized images with no pretraining on visual tasks whatsoever. Specifically, pass an image to an encoder-decoder visual model trained in a completely unsupervised manner on images -> take the encoded representation -> pass the encoded representation to an LLM as tokens -> get accurate captions. The tests were done on gpt-j, which is not multimodal and only has about 7bn params. The only caveat is that a linear mapping model needs to be trained to map the vector space from the encoder-decoder model to the embedding space of the language model, but this isn't doing any conceptual labour, it's only needed to align the completely arbitrary coordinate axes of the vision and language models, which were trained separately (akin to an American and a European to agreeing to use metric or imperial — neither’s conception of the world changes).
It's not intuitive, but it's hard to argue with these results. Even small LLMs can caption images. Sure, they don't get the low-level details like the texture of grass, but they get the gist.
I keep reading your sort of analysis, but honestly, those priors need updating. I had to update when learning this. If 7bn params can do it, 175bn params with multimodality can certainly do it.
It's true that humans need symbol grounding, but we don't see hundreds of billions of sequences. There are theoretical reasons (cf category theory) why this could work, albeit probably limited to gist rather than detail.
The question is whether an assembly of components, one of which is an LLM (others would include memory and whatever else is needed to make it a self-contained loop with a notion self-identity) can reason.
You’re assuming that reasoning isn’t just pattern matching.
Pattern matching in LLMs is specifically at the language level, as in "predict the next token in a stream", I should have probably specified this bit.
You seem to refer to a more general kind of pattern matching when you say, where we're looking at a complicated structural model within which humans reason about the behavior of different system components and their interactions.
There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind.
If all you have is language, and predictive text, there is no way to use this to simulate and predict the rules in a higher-dimensional space than "stream of tokens", or let's put it another way: there would need to be an extremely high correlation between the pattern in the problem space and in language space for the likelihood of the model guessing right to be anywhere near human levels.
You claim this as if it is a foregone conclusion yet it is anything but. This is an active area of study, even within human cognition https://en.m.wikipedia.org/wiki/Predictive_coding
A complex system clearly does not need to be designed to evoke some behavior. Just look to the concept of emergent phenomenon. See evolution through natural selection as an obvious counter example.
Evolution selects for fitness and not reasoning ability. Yet here we are.
It is not known if the same isn't happening now for the transformer model.
LLMs are universal function approximators. If reasoning ability is at all representable through such a process, it is possible for LLMs to find it.
https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it.
Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back.
This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems.
Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function.
If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.
And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses.
And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.
Yes, in the general case, you can make a model for most types of logic problems (but be careful there, there are real problems you couldn't represent as "a bunch of numbers" and a single transformation), the issue is that the model cannot build that representation even for simple cases because it,'s used to operate at on language (and maybe some other fuzzy input type), which has to be _reliably_ interpreted to this number space; in different ways for different problems; without breaking the language processing.
You're aware multimodel transformers do exactly this?
The functions also have to be representable in a way that the model can transform them usefully. If the model does not have the capability to transform into a different arbitrary space for every separate sentence, reasoning is not possible.
Even then, reasoning would IMO be limited by other representation effects. But maybe we could have a discussion at that point.
I just don't think transformers are going to get us to an intelligent agent, which is completely ok, they are a useful ML tool that we can use to create somewhat accurate models for complicated systems, and that's what they're designed for.
You're making an authoritative statement on how reasoning works based on conjecture. I remind you that this field is in active study.
The "function" that sufficiently approximates the phenomenon of higher order reasoning may well mode collapse into a compact representation that need not exhaustively span the space of all "sentence representations".
You can illustrate for yourself that it is not at all clear your premise is a necessary condition.
Our brains can reason. Are you proposing its mechanism of action is by "transformation into a different arbitrary space for every sensory experience X"?
I mean maybe? But what if there exists some higher order space that actually generalizes perfectly and captures the basis vectors of these seemingly orthogonal spaces? That can also be the case. Which is why in truth we don't yet know, and to say your view is correct is suspect.
Hope you see now why this claim is an overstatement and missing some qualifiers.
There isn't really any other feasible option.
This is what humans do, we visualize and predict the outcome of actions within a system. And if our imagination is not enough, we can outsource the reasoning to the outside world, by doing experiments or using tools like geometry etc. (drawing in the sand, drawing on paper).
It is impossible to arrive at a conclusion without "doing the work", unless you've seen the result before and can recite it.
Therefore, you need to be able to model the problem domain in order to solve the problem.
They have been fed incredible amounts of data which allows them to stochastically parrot really complex things, this makes people think they're intelligent. Designing a test that actually tests intelligence rather than parroting ability is tricky, but whenever people do they fail them.
I just logged into ChatGPT and asked it repeatedly to generate a novel thought experiment intended to prove that LLMs are incapable of reasoning. I kept trying to refine it by explaining what was wrong with each suggestion.
It completely failed. It just spat out variations of well known dilemmas or paradoxes, even when I specifically asked it to create something original it just made nonsense.
It's exactly like the crypto hype wave. Everyone dreaming up semi-plausible futures, based on a whole chain of unfounded assumptions.
We put a camera on a multimodal LLM, it interprets the visual world and sees before it a bunch of blocks. It looks at the task list it has that says "pick up red blocks, put them in blue bin". The visual components identifies the red blocks and the textual component issues commands to its drive unit which calculates the best path and how to use it's manipulators to pick up the blocks.
This is a very basic chain of existence. We have world identification, we have actuation on motivation, we have interaction with the environment. We can do this now. These goals have already been achieved. Companies are already testing more complex models with much more general instructions such as "Pick up everything you think is trash" or "Organize this room" to see the emergent behaviors that come out of the models.
You are seemingly a few years behind what has already been done, and why people are starting to get concerned.
"Consciousness" has ~22 different definitions; some are easily met, some are not even met by the humans giving those definitions (for example, I've seen people here on HN who write as if consciousness can somehow magically solve the halting problem).
There's only one test for it in AI, that I've heard of, and that I think is worth exploring: https://www.youtube.com/watch?v=LWf3szP6L80
So there should be absolutely no surprise that, when something comes along that behaves in certain ways very much like a human (and unlike anything else), it is assumed to share the basic qualities of such. It is how we judge other people, after all - it's not like you do a brain scan to determine whether someone you are talking to is "sentient", whatever that means to you. You judge that by their behavior, modelling what a "sentient" would do and comparing the result to what they actually do.