My rule of thumb is to take every single LLM prompt and just imagine that it's prefixed with:
"Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."
"Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."
>It has a relationship with reality insofar as one exists within the training data
This is true of anything that learns.
Sure, but most things that learn have actual reality as a training set. LLMs have human curated data, which isn’t and can’t be perfectly representative of reality.
Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story.
LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects, actions, and ideas.
They can reasoning, not just mimic: LLMs can perform logical reasoning, using their internal knowledge base to solve problems or answer questions. This might involve following multi-step instructions, drawing inferences from information provided, or adapting to new prompts in a way that requires a degree of abstract thinking.
Beyond simple probabilities: Yes, LLMs do consider the probability of certain word sequences, but their output is far more sophisticated than just picking the most likely next word. They weigh context, concepts, relationships, nuance, logic, and even the unstated but inferred purpose of the user when generating responses.
The LLM neural network contains a semantic model and it performs some type of reasoning over that model. The idiot and the genius both can see that ChatGPT has some reasoning capability.
https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf
"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.
By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.
This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"
Here is a succinct example using Claude:
Prompt:
Using the decimal system, redefine all symbols by shifting them one to right such that 0 becomes 1, 1 becomes 2, 2 becomes 3, etc. Now, let's do math. What is 2x3?
===
Okay, with the symbols shifted one to the right in the decimal system:
0 becomes 1
1 becomes 2
2 becomes 3
3 becomes 4
4 becomes 5
5 becomes 6
6 becomes 7
7 becomes 8
8 becomes 9
9 becomes 0 (looping back around)
So if 2 becomes 3, and 3 becomes 4, then:
2 x 3 = 3 x 4 = 12
Shifting those digits one to the right based on the new mapping: 12 becomes 23
Therefore, under this new symbol mapping, 2 x 3 = 23
===
Regardless, if you are interpreting 2x3 to be in the original space, then 2x3 = 6 which would be 7 in the new space. If you are interpreting 2x3 to be in the new space (which is how I would interpret it), then 2x3 is 1x2 = 2 which is 3.
I think it's clear from this example that the LLM has 0 ability to reason.
It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or machine learning understands this.
By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.
> By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.
And therein lies the problem with this whole debate. I think a huge part of the debate is conflating the fact that most humans do not reason well (I wouldn't say they cannot reason) and make mistakes means that reasoning is something fuzzy and make statements like "LLMs reason about as well as humans". Very few humans outside of mathematicians practice logic on a daily basis. Most humans get by with muscle memory and pattern recognition of previous tasks. Just because LLMs are roughly as good as humans at this behavior does not make them able to reason. I would be totally fine if people just replaced "can reason" with "are useful" within their statements so they would look more like "LLMs are as useful as humans in answering MCAT tests." To imply there is a rational actor deriving responses from first order logic is disingenuous in my opinion.
There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate. For the record, I use neural networks every day and believe that they are incredibly useful and can be purpose-built to beat humans on a large set of tasks. Can they reason? No. Can formal theorem provers reason? No they cannot. They can only verify.
The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent.
Solving natural language processing tasks requires an understanding of the context of words, and this contextual understanding is constructed through a form of reasoning. You can choose to ignore all of the reasoning that they are demonstrably capable of, and narrowly focus on their imperfections, then declare they are capable of "0 reasoning", but that conclusion is an oversimplification. With actual "0 reasoning" ability they wouldn't be able to perform standard NLP tasks such as translation or summarization anywhere close to their current level of accuracy.
The Wikipedia definition refers to consciousness, so that excludes machines. That informal definition is not useful in the context of AI. We need to use technical definitions. Otherwise it is subjective and we end up wasting time squabbling about semantics.
I agree. I more or less disagree with all your other claims. A linear model is not reasoning when it extrapolates. Neural networks do not "apply abstract rules of logical reasoning to problems outside of their training set".
> Solving natural language processing tasks requires an understanding of the context of words
Agree.
> and this contextual understanding is constructed through a form of reasoning
Disagree. This models are predictive models. Here is a simple estimator for your height: ((mom's height)+(dad's height))/2. This estimator is very good and modern genomic techniques struggle to beat it. Does this model reason? Obviously not. Yet, the 1B parameter genomics models do reason about the genes? That makes no sense. They are both models. Incapable of reason.
> You can choose to ignore all of the reasoning that they are demonstrably capable of
They have not demonstrated any reasoning. Just because they can solve olympiad level math questions, it does not mean they can reason. This does not take away from the power of these models. These models will have full conversations about reasoning, explain syllogistic logic, and apply different deductions which seemingly show they are reasoning. It's all impressive. Then you are harshly reminded by the following interaction that it is just an amazing ability to memorize:
Again with Claude. Prompt:
A farmer and a goat need to cross a river. There is a boat that can fit exactly one human and one animal. What is the least number of crossings needed to get everything to the other side of the river?
===
Okay, let's approach this classic logic puzzle systematically:
Given:
- There is a farmer, a goat, and a boat that can carry one human and one animal at a time.
- The farmer cannot leave the goat alone with the crop, as the goat will eat the crop.
- The objective is to get the farmer, goat, and the boat across the river using the minimum number of crossings.
To solve this, we need to find a sequence of crossings that adheres to the constraint and achieves the objective in the fewest steps.
Here's one valid solution that requires 7 crossings:
1) Farmer rows the goat across the river.
2) Farmer rows back alone.
3) Farmer rows the crop across the river.
4) Farmer rows the goat back.
5) Farmer rows across again.
6) Goat rows across.
7) Farmer rows the goat across.
Explanation:
- Crossing 1 gets the goat across initially.
- Crossing 2 brings the farmer back to pick up the crop.
- Crossing 3 takes the crop across.
- Crossing 4 brings the goat back so it's not alone with the crop.
- Crossing 5 takes the farmer across.
- Crossing 6 takes the goat across since it's now safe.
- Crossing 7 takes the farmer and goat together to the desired side.
This sequence ensures the goat is never left unattended with the crop, adhering to the constraint.
It can be proven that 7 crossings is the minimum required, as:
- The goat must cross twice (there and back).
- The crop must cross once.
- The farmer must cross at least twice to accompany the goat and crop.
- Additional crossings are needed to reposition the boat.
So the optimal solution involves 2 crossings for the goat, 1 for the crop, and a minimum of 4 additional crossings for the farmer and boat repositioning, totaling 7 crossings.
Therefore, the least number of crossings needed to get the farmer, goat, and boat across the river is 7.
===
All appearances of reasoning are nullified by a response that pulls stuff out of thin air and states it confidently.
If you can prove this is wrong then you should publish your result.
https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf
"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.
By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.
This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"
First, your summary of the paper is nowhere to be found in the paper so I assume this is your summary. You say "the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training." This paper does not include training details. It delegates it to another paper in [10]. From the details in this paper, it is trained on 100 out of the 104 total relations. However, there are only 12 distinct relations: mother, husband, wife, son, daughter, uncle, aunt, brother, sister, nephew, niece. That means, each relation is seen ~8 times. Now, your claim is "underlying logical patterns and reason about new family tree instances not seen during training" but that's a gross misrepresentation of what is happening here. First, it's given multiple instances of the same tree with different labels. Second, the inputs appear to be the 24 people involved and so you cannot possibly extend this to new tree topologies. Finally, this to me is the money quote of the paper:
> Does it make use of the isomorphism between the two family trees to allow it to encode them more efficiently and to generalize relationships in one family tree by analogy to relationships in the other? If it does all these things, it seems reasonable to say that it is doing inference rather than mere association.
Now, we have to be careful here because inference might be construed as reasoning. Obviously, the model is performing some type of statistical inference where a model has been posited (3 layer neural network) and the output is being trained (presumably - no training details) to minimize classification error through something like KL which is equivalent to MLE and so it is indeed a statistical inference. This model is so simple, you could manually work out the inference by doing a page full of multiplications. I brought this up before, so I'll ask you to specifically address this point. No one claims linear models perform reasoning. Why are you proposing that this 3 layer (read, 3 matrix multiplies) is doing reasoning?
https://www.cs.toronto.edu/~hinton/coursera/lecture4/lec4.pd...
> No one claims linear models perform reasoning. Why are you proposing that this 3 layer (read, 3 matrix multiplies) is doing reasoning?
A 3 layer neural network is a non-linear function. It is not a linear model. There are activation functions between the layers which make it non-linear.
Picking the most likely next word is an extremely sophisticated algorithm, if you could do that you would almost sound human. Like an LLM!
However, focusing on just picking the most likely next words do cause some issues, such as you being biased towards words that were posted instead of those that were too boring to post, so there are still things that such an algorithm can't understand like the probabilities since those are so unevenly posted.
Neural networks pre-trained on data for a completely different task, learn new tasks much faster. With a GPT-like transformer, you can feed it PCM audio samples encoded as uuencoded text, or paintings encoded in the same way, and it learns how to translate English <-> Russian when later trained on that, much faster than from a completely randomized model. There's something common to those seemingly disparate tasks that is learned. "Abstraction" may be the right word for this.
A more parsimonious hypothesis is that random networks start out broken, structurally incapable of computation because the structure has parts where information stops flowing or signal gain is so low at certain choke points that it’s presence is like a random coin flip.
Training the network to compute ANYTHING fixes this flow problem, making subsequent training easier, without introducing any kind of abstraction.
https://twitter.com/colin_fraser/status/1785132544482226679
I just tried a similar question now with ChatGPT4:
"If a man and a goat are on one side of a river, what is the minimum amount of trips required to get the man and goat to the other side in a boat. Assume the boat can hold at most one animal and one human."
ChatGPT: 3 trips
That is very much closer to "trying to predict next word from examples" than "billion-dollar model with internal reasoning".
I tried a few chain of thought prompts for the original question and GPT-3.5 was sometimes (randomly) able to find the correct answer on the first attempt for this one
https://chat.openai.com/share/c144ba23-2f78-4cc8-a1c5-ca3106...
Instructions:
1. Do not include any assumptions that I have not mentioned here.
2. Before solving the problem, state the goal of the problem.
3. After each step of your reasoning, state where the man and the goat are now standing, and state if the goal has been achieved or not, if the goal has been achieved then stop. If the goal has not been achieved then explain why not.
Spoilage is incredibly easy to do. It is about information leakage and you have to think very carefully about how information can leak through in subtle ways. Specifically #1 and #2 are strong hints that there is a trick to the problem (i.e. is this something you would use in a generic prompt?). #3 is a reiteration of the problem, that gives extra weight. You can decrease the weight by restating as "state where the man and any animals are located" (notice there's lower information gain here). " if the goal has been achieved then stop." is a big hint. To reason, it should know when to stop.
I posted some recent river crossing tweets in this comment that may be of interest to you https://news.ycombinator.com/item?id=40231409
The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. These models are primarily trained for solving natural language processing tasks, and so they are predisposed to answer in a certain way through training and tuning. The models produce less verbose output by default to reduce cost (each token costs money). Telling the model to generate more tokens in step-by-step reasoning enables it to "think" further as it can only "think" when generating each token.
OpenAI could train or tune ChatGPT to "spoil" itself by default when answering any problem that it identifies as a logic problem. It is somewhat arbitrary.
I think you're missing a bit here. Look at the middle tweet where the person constructed it fail the logic. There are no tricks. What you're missing is the signal you're giving it, how it is spoiling the question in a subtle way. That's very different that a reasoning machine. We can't trust it to reason if it can only "reason" when we give it explicit instructions to do so that do not generalize for many tasks. That's not really reasoning...
> OpenAI could train or tune ChatGPT to "spoil" itself by default
They have and it's provable
They are trained a certain way to perform specific types of tasks, primarily natural language processing tasks. They have necessarily learned some methods of reasoning in order to do what they were trained to do. No one is pretending that these are symbolic logic mechanical theorem provers. They are tuned a certain way to respond in a specific manner and they only do what they are told. If you want it to use reasoning then you need to tell it to use reasoning. It's a chat bot running on a neural network and it is not self aware.
Hopefully the next generation of AI will be more reasonable. We are not there yet.
GPT4: https://chat.openai.com/share/1beb5391-d321-4515-945e-38233f...
She's a bit older now (5) but e.g. a few days ago I was talking about cleaning the whole house. She said "you didn't clean the WHOLE house, look there's something you didn't clean".
I'm still waiting for this. Here's some recent tweets about people with variations of the river crossing puzzle. All LLMs seem to fail pretty badly.
LLMs are impressive enough as they are (compression systems with human language interface), you don't need to hype them up to something they're not.
https://twitter.com/jeremyphoward/status/1783712611126964627
https://twitter.com/WaltonStevenj/status/1785145923771011215