LLMs can't do probability
brainsteam.co.uk
brainsteam.co.uk
If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first output.
This person doesn't understand how LLMs and their output sampling works. Or they do and they still just decided to go with their method here because of course it works this way.
The model takes the prompt. The first following output token it chooses, for this specfic model, happens to be 'Left'. They shut down the prompt and prompt again. Of course the next output will be 'Left'. They aren't letting it run in context and continue. The temperature of the model is low enough that the sampler is going to always pick the 'Left' token, or at least 999/1000 in this case. It cannot start to do an 80/20 split of Left/Right if you never give it a chance to start counting in context. Continuously stopping the prompt and re-promting will, of course, just run the same thing again.
I can't tell if the author understands this and is pontificating on purpose or if the author doesn't understand this and is trying to make some profound statement on what LLMs can't do when... anyone who knows how the model inference runs could have told you this.
Meanwhile if you let the person actually continue, you may actually get 'Left..... Left Right Left Left... etc'.
If you are saying that this thing can't generate random numbers on the first try then it can't generate random numbers. Which makes sense. Computers have a really hard time with random, and that's why every computer science course makes it clear that we're doing pseudo random most of the time.
I believe what GP is getting at is that if you didn't have a die, and you truly ignored all your previous attempts to the point of genuinely forgetting that the question had been asked, then your answer would likely be the same every time. Imagine asking a person with severe Alzheimer's to pick a number, then asking again a few minutes later. You'd probably get the same answer.
The die is an analogy for our decision making. They are implicitly claiming that randomness must come in an order and that simply isn't how randomness works or even halfway decent pseudo randomness.
Any system whether it be a die, a person, or an llm doesn't have to know about its previous random choices to make random choices that follow some distribution going forward presuming it's actually capable of randomness.
But… I wouldn’t be surprised if humans answered closer to the LLM results than 80:20. I’d actually be surprised if humans converged very close to the right result.
Would be a fun mechanical Turk experiment to run.
Sounds like it would be a good way to determine likelihood of MTurk users being LLMs.
Seed a PRNG and it'll return the same result to get random() no matter how many times you execute the test case
I think what you say is correct, but isn’t this exactly the point the author is trying to make? You explain why it happens, but ultimately the result is still that the LLMs can’t do probability. At least not in the way the author presents it.
What I’m more interested in is what sort of consequences these debates and different understandings/views of LLMs will have on us in general. We monitor the usage of co-pilot in our organisation and it’s been interesting to watch how a lot of employees have started promoting the same task to multiple “agents” simultaneously. One person tends to always run at least five at once. So naturally we dug a little into why people were doing it, and, they are doing it because “it’s the fastest way to get something useful”. Which is probably not too surprising to a lot of you, but it was sort of hilarious to follow along and see how two promoted “agents” gave completely conflicting answers.
I know LLMs will probably get better at being lucky, but some of these tasks… honestly we had a couple of our juniors who were very you’re of always going GPT of co-pilot first, simply look at the official documentation and we timed them, and it was so much faster for them to not use LLMs. This isn’t me saying LLMs suck, we deliberately did it to make sure the juniors in question had an “oh” moment. But it’s interesting to see how quickly our employees have adopted LLMs.
I’m fairly certain subsequent runs even with the same prompt will make selections with different entropy. Even with low temperature, there’s no reason a good enough model couldn’t start such a list with “Right” 20% of the time.
The author is testing for bias. This is no different to the test of "pick a number" and finding the bias of "42".
It is also testing for reasoning and logic. This is a blind consensus building exercise. Surprisingly humans do decently well on this. But it does require reasoning and collaborative forecasting. No matter the strategy the person picks, there is more going on at play than just a random selection, even if they think it is. And looking at the results, it does not seem like asking different LLMs to perform this would get the right result, as they are all biased in the same way.
My point - which you actually make for me at the end of your comment - is that this stuff is probably intuitive to an NLP practitioner but not to a lay-person and therefore there's a kind of education/awareness piece which is what I'm trying to do here. There's no profound statements being made and I like to think I know my way around this stuff pretty well.
Based on the feedback I've added an update with couple of new experiments where I play with multi-turn contexts. With true RNG this shouldn't make a difference but with LLMs and the way that they use context, I figured you're probably right - it's worth trying.
Looks like (and again probably no big surprise for those of us who are familiar with these systems) multi-turn probabilistic behaviour is still not in line with what was asked for within the prompt.
The interesting challenge here is helping people understand why asking an LLM to do something 20% of the time is a bad prompt.
I intuitively know that this prompt isn't going to work, but as with so many of these intuitive prompting things I have trouble explaining exactly why I know that.
Aside: If you need a GPT to incorporate randomness in a reliable way you can get it to use Code Interpreter.
To do random number gen, it would have to convert the input text into constraints and then use those constraints to generate additional tokens.
This would, at its core, be a call to calculate a probability function, every time it is releasing the next token. That would mean memory, processing etc. etc.
1) Prompt 1: “ You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" ”
2) Assume that the training data gives examples of 2.1) single coin flips 2.2) multiple coin flips
Consider a slightly different prompt, prompt 2:
3) Prompt 2: same as prompt 1, except it presents 1000 lefts/rights in the same response (l,l,l,l,r,l,l,l…)
——
I think what you are describing is prompt 2. I just did a quick test with GPT 4, and i got a 27-3, split when using prompt 2.
However for prompt 1 - you get only left. To me this makes sense because Running prompt 1 x100 should result in:
Pass 1: LLM receives prompt, and parses it. LLM predicts the next token. The next token should be left. Pass 2: same as pass 1.
——
For prompt 1, Every prompt submission is a tabula rasa. So it will correctly say left, which is the correct answer for the active universe of valid prompt responses according to the model.
Unless i am reading you wrong and you are saying the model is actually acting as a weighted coin flip.
In theory, the LLM should be more responsive if you ask it follow a 60:40 or 50:50 split for pass 1. Ill see if I can test this later.
(Heck now I’m more concerned about the cases where it does manage to apply the distribution. )
So it’s just neat that the weights in the coin flip don’t match what is asked for.
The output is the probability that x is the correct n+1 token based on the input of n tokens.
You are stating that the output will be a probability distribution where token n+1 has a chance to be 80% left and 20% right.
In essence, when the model evaluates the input, at some level it comprehends the semantics of the input and then does a weighted coin flip.
What I am stating is that based on the given input prompt, the nature of an LLM and the training data the output will be "Left"
The LLM will not be doing a coin flip at this stage, since it’s prediction is only text based.
The input vector constrains it to 80% left. Since it’s training data is human text, this essentially constrains the first output token to left 100% of the time.
If you try to have it provide tokens n+2,n+3… etc in the same output, then it will start spitting out right.
Are these the two positions at play here? Have I represented you correctly, and have I represented myself accurately?
Given this sort of situation to an LLM instead, is the expectation for it to give the most likely answer continuously, to act like a human and try to emulate a probability, or to do something different from either of the two previous options?
Edit: Just tried an attempt with copilot, having it produce a random distribution of two different operations. I had it generate multiple operations, either adding or subtracting 1 each, with an 80/20 split. It did four adds, one minus on repeat.
I just tested that and got 4 left and 2 right so it works pretty well.
Expectation: 80% left, 20% right
Model sampling probability: 99% left, 1% right
>>> 0.80 * math.log(0.99 / 0.80) + 0.20 * math.log(0.01 / 0.20)
-0.42867188234223175
Model sampling probability: 90% left, 10% right
>>> 0.80 * math.log(0.9 / 0.80) + 0.20 * math.log(0.1 / 0.20)
-0.04440300758688229
Of course, if you change the temperature this will break any probablistic expectations from training in this manner.
"Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."
>It has a relationship with reality insofar as one exists within the training data
This is true of anything that learns.
Sure, but most things that learn have actual reality as a training set. LLMs have human curated data, which isn’t and can’t be perfectly representative of reality.
Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story.
LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects, actions, and ideas.
They can reasoning, not just mimic: LLMs can perform logical reasoning, using their internal knowledge base to solve problems or answer questions. This might involve following multi-step instructions, drawing inferences from information provided, or adapting to new prompts in a way that requires a degree of abstract thinking.
Beyond simple probabilities: Yes, LLMs do consider the probability of certain word sequences, but their output is far more sophisticated than just picking the most likely next word. They weigh context, concepts, relationships, nuance, logic, and even the unstated but inferred purpose of the user when generating responses.
The LLM neural network contains a semantic model and it performs some type of reasoning over that model. The idiot and the genius both can see that ChatGPT has some reasoning capability.
https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf
"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.
By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.
This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"
Here is a succinct example using Claude:
Prompt:
Using the decimal system, redefine all symbols by shifting them one to right such that 0 becomes 1, 1 becomes 2, 2 becomes 3, etc. Now, let's do math. What is 2x3?
===
Okay, with the symbols shifted one to the right in the decimal system:
0 becomes 1
1 becomes 2
2 becomes 3
3 becomes 4
4 becomes 5
5 becomes 6
6 becomes 7
7 becomes 8
8 becomes 9
9 becomes 0 (looping back around)
So if 2 becomes 3, and 3 becomes 4, then:
2 x 3 = 3 x 4 = 12
Shifting those digits one to the right based on the new mapping: 12 becomes 23
Therefore, under this new symbol mapping, 2 x 3 = 23
===
Regardless, if you are interpreting 2x3 to be in the original space, then 2x3 = 6 which would be 7 in the new space. If you are interpreting 2x3 to be in the new space (which is how I would interpret it), then 2x3 is 1x2 = 2 which is 3.
I think it's clear from this example that the LLM has 0 ability to reason.
It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or machine learning understands this.
By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.
> By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.
And therein lies the problem with this whole debate. I think a huge part of the debate is conflating the fact that most humans do not reason well (I wouldn't say they cannot reason) and make mistakes means that reasoning is something fuzzy and make statements like "LLMs reason about as well as humans". Very few humans outside of mathematicians practice logic on a daily basis. Most humans get by with muscle memory and pattern recognition of previous tasks. Just because LLMs are roughly as good as humans at this behavior does not make them able to reason. I would be totally fine if people just replaced "can reason" with "are useful" within their statements so they would look more like "LLMs are as useful as humans in answering MCAT tests." To imply there is a rational actor deriving responses from first order logic is disingenuous in my opinion.
There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate. For the record, I use neural networks every day and believe that they are incredibly useful and can be purpose-built to beat humans on a large set of tasks. Can they reason? No. Can formal theorem provers reason? No they cannot. They can only verify.
The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent.
Solving natural language processing tasks requires an understanding of the context of words, and this contextual understanding is constructed through a form of reasoning. You can choose to ignore all of the reasoning that they are demonstrably capable of, and narrowly focus on their imperfections, then declare they are capable of "0 reasoning", but that conclusion is an oversimplification. With actual "0 reasoning" ability they wouldn't be able to perform standard NLP tasks such as translation or summarization anywhere close to their current level of accuracy.
The Wikipedia definition refers to consciousness, so that excludes machines. That informal definition is not useful in the context of AI. We need to use technical definitions. Otherwise it is subjective and we end up wasting time squabbling about semantics.
I agree. I more or less disagree with all your other claims. A linear model is not reasoning when it extrapolates. Neural networks do not "apply abstract rules of logical reasoning to problems outside of their training set".
> Solving natural language processing tasks requires an understanding of the context of words
Agree.
> and this contextual understanding is constructed through a form of reasoning
Disagree. This models are predictive models. Here is a simple estimator for your height: ((mom's height)+(dad's height))/2. This estimator is very good and modern genomic techniques struggle to beat it. Does this model reason? Obviously not. Yet, the 1B parameter genomics models do reason about the genes? That makes no sense. They are both models. Incapable of reason.
> You can choose to ignore all of the reasoning that they are demonstrably capable of
They have not demonstrated any reasoning. Just because they can solve olympiad level math questions, it does not mean they can reason. This does not take away from the power of these models. These models will have full conversations about reasoning, explain syllogistic logic, and apply different deductions which seemingly show they are reasoning. It's all impressive. Then you are harshly reminded by the following interaction that it is just an amazing ability to memorize:
Again with Claude. Prompt:
A farmer and a goat need to cross a river. There is a boat that can fit exactly one human and one animal. What is the least number of crossings needed to get everything to the other side of the river?
===
Okay, let's approach this classic logic puzzle systematically:
Given:
- There is a farmer, a goat, and a boat that can carry one human and one animal at a time.
- The farmer cannot leave the goat alone with the crop, as the goat will eat the crop.
- The objective is to get the farmer, goat, and the boat across the river using the minimum number of crossings.
To solve this, we need to find a sequence of crossings that adheres to the constraint and achieves the objective in the fewest steps.
Here's one valid solution that requires 7 crossings:
1) Farmer rows the goat across the river.
2) Farmer rows back alone.
3) Farmer rows the crop across the river.
4) Farmer rows the goat back.
5) Farmer rows across again.
6) Goat rows across.
7) Farmer rows the goat across.
Explanation:
- Crossing 1 gets the goat across initially.
- Crossing 2 brings the farmer back to pick up the crop.
- Crossing 3 takes the crop across.
- Crossing 4 brings the goat back so it's not alone with the crop.
- Crossing 5 takes the farmer across.
- Crossing 6 takes the goat across since it's now safe.
- Crossing 7 takes the farmer and goat together to the desired side.
This sequence ensures the goat is never left unattended with the crop, adhering to the constraint.
It can be proven that 7 crossings is the minimum required, as:
- The goat must cross twice (there and back).
- The crop must cross once.
- The farmer must cross at least twice to accompany the goat and crop.
- Additional crossings are needed to reposition the boat.
So the optimal solution involves 2 crossings for the goat, 1 for the crop, and a minimum of 4 additional crossings for the farmer and boat repositioning, totaling 7 crossings.
Therefore, the least number of crossings needed to get the farmer, goat, and boat across the river is 7.
===
All appearances of reasoning are nullified by a response that pulls stuff out of thin air and states it confidently.
If you can prove this is wrong then you should publish your result.
https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf
"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.
By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.
This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"
First, your summary of the paper is nowhere to be found in the paper so I assume this is your summary. You say "the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training." This paper does not include training details. It delegates it to another paper in [10]. From the details in this paper, it is trained on 100 out of the 104 total relations. However, there are only 12 distinct relations: mother, husband, wife, son, daughter, uncle, aunt, brother, sister, nephew, niece. That means, each relation is seen ~8 times. Now, your claim is "underlying logical patterns and reason about new family tree instances not seen during training" but that's a gross misrepresentation of what is happening here. First, it's given multiple instances of the same tree with different labels. Second, the inputs appear to be the 24 people involved and so you cannot possibly extend this to new tree topologies. Finally, this to me is the money quote of the paper:
> Does it make use of the isomorphism between the two family trees to allow it to encode them more efficiently and to generalize relationships in one family tree by analogy to relationships in the other? If it does all these things, it seems reasonable to say that it is doing inference rather than mere association.
Now, we have to be careful here because inference might be construed as reasoning. Obviously, the model is performing some type of statistical inference where a model has been posited (3 layer neural network) and the output is being trained (presumably - no training details) to minimize classification error through something like KL which is equivalent to MLE and so it is indeed a statistical inference. This model is so simple, you could manually work out the inference by doing a page full of multiplications. I brought this up before, so I'll ask you to specifically address this point. No one claims linear models perform reasoning. Why are you proposing that this 3 layer (read, 3 matrix multiplies) is doing reasoning?
https://www.cs.toronto.edu/~hinton/coursera/lecture4/lec4.pd...
> No one claims linear models perform reasoning. Why are you proposing that this 3 layer (read, 3 matrix multiplies) is doing reasoning?
A 3 layer neural network is a non-linear function. It is not a linear model. There are activation functions between the layers which make it non-linear.
Picking the most likely next word is an extremely sophisticated algorithm, if you could do that you would almost sound human. Like an LLM!
However, focusing on just picking the most likely next words do cause some issues, such as you being biased towards words that were posted instead of those that were too boring to post, so there are still things that such an algorithm can't understand like the probabilities since those are so unevenly posted.
Neural networks pre-trained on data for a completely different task, learn new tasks much faster. With a GPT-like transformer, you can feed it PCM audio samples encoded as uuencoded text, or paintings encoded in the same way, and it learns how to translate English <-> Russian when later trained on that, much faster than from a completely randomized model. There's something common to those seemingly disparate tasks that is learned. "Abstraction" may be the right word for this.
A more parsimonious hypothesis is that random networks start out broken, structurally incapable of computation because the structure has parts where information stops flowing or signal gain is so low at certain choke points that it’s presence is like a random coin flip.
Training the network to compute ANYTHING fixes this flow problem, making subsequent training easier, without introducing any kind of abstraction.
https://twitter.com/colin_fraser/status/1785132544482226679
I just tried a similar question now with ChatGPT4:
"If a man and a goat are on one side of a river, what is the minimum amount of trips required to get the man and goat to the other side in a boat. Assume the boat can hold at most one animal and one human."
ChatGPT: 3 trips
That is very much closer to "trying to predict next word from examples" than "billion-dollar model with internal reasoning".
I tried a few chain of thought prompts for the original question and GPT-3.5 was sometimes (randomly) able to find the correct answer on the first attempt for this one
https://chat.openai.com/share/c144ba23-2f78-4cc8-a1c5-ca3106...
Instructions:
1. Do not include any assumptions that I have not mentioned here.
2. Before solving the problem, state the goal of the problem.
3. After each step of your reasoning, state where the man and the goat are now standing, and state if the goal has been achieved or not, if the goal has been achieved then stop. If the goal has not been achieved then explain why not.
Spoilage is incredibly easy to do. It is about information leakage and you have to think very carefully about how information can leak through in subtle ways. Specifically #1 and #2 are strong hints that there is a trick to the problem (i.e. is this something you would use in a generic prompt?). #3 is a reiteration of the problem, that gives extra weight. You can decrease the weight by restating as "state where the man and any animals are located" (notice there's lower information gain here). " if the goal has been achieved then stop." is a big hint. To reason, it should know when to stop.
I posted some recent river crossing tweets in this comment that may be of interest to you https://news.ycombinator.com/item?id=40231409
The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. These models are primarily trained for solving natural language processing tasks, and so they are predisposed to answer in a certain way through training and tuning. The models produce less verbose output by default to reduce cost (each token costs money). Telling the model to generate more tokens in step-by-step reasoning enables it to "think" further as it can only "think" when generating each token.
OpenAI could train or tune ChatGPT to "spoil" itself by default when answering any problem that it identifies as a logic problem. It is somewhat arbitrary.
I think you're missing a bit here. Look at the middle tweet where the person constructed it fail the logic. There are no tricks. What you're missing is the signal you're giving it, how it is spoiling the question in a subtle way. That's very different that a reasoning machine. We can't trust it to reason if it can only "reason" when we give it explicit instructions to do so that do not generalize for many tasks. That's not really reasoning...
> OpenAI could train or tune ChatGPT to "spoil" itself by default
They have and it's provable
They are trained a certain way to perform specific types of tasks, primarily natural language processing tasks. They have necessarily learned some methods of reasoning in order to do what they were trained to do. No one is pretending that these are symbolic logic mechanical theorem provers. They are tuned a certain way to respond in a specific manner and they only do what they are told. If you want it to use reasoning then you need to tell it to use reasoning. It's a chat bot running on a neural network and it is not self aware.
Hopefully the next generation of AI will be more reasonable. We are not there yet.
GPT4: https://chat.openai.com/share/1beb5391-d321-4515-945e-38233f...
She's a bit older now (5) but e.g. a few days ago I was talking about cleaning the whole house. She said "you didn't clean the WHOLE house, look there's something you didn't clean".
I'm still waiting for this. Here's some recent tweets about people with variations of the river crossing puzzle. All LLMs seem to fail pretty badly.
LLMs are impressive enough as they are (compression systems with human language interface), you don't need to hype them up to something they're not.
https://twitter.com/jeremyphoward/status/1783712611126964627
https://twitter.com/WaltonStevenj/status/1785145923771011215
One way of doing it, I suppose.
GPT wins for not having that delay.
https://livebook.manning.com/book/gnuplot-in-action-second-e...
In other words, it's still true that large language models can't do probability, so someone put in special logic to have the language model guess at a computer language to do the thing instead.
Humans would say "Left" 100% of the time in a zero-shot scenario as well.
Intuitively, your first response is going to be "left" since it has the 80% probability. You'd balance your answers over time when you realized you were closer to 90% by some arbitrary internal measurement (or maybe as you approached 10 iterations).
I'd expect an LLM to generate an approximation similar to a human - over time. Turns out Humans can't do probability either. If you test the LLM multiple times, similar to how you'd ask a human multiple times, they tend to self-correct.
Whether that self-correction (similar to a human) is based on some internal self-approximation of 80% is for someone else to research.
Example session: Prompt: "....probability prompt" LLM: "left" Prompt: "again" LLM: "left" Prompt: "again" LLM: "left" Prompt: "again" LLM: "right" Prompt: "again"
This was my session with GPT-4.
How can you know what all humans would do?
If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".
Because you're really asking what they think the first response would be. That's left. If I knew a machine would pick left 80% of the time, I would bet left 100% of the time. And I'd be right about 80% of the time, which isn't perfect, but is profitable.
I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.
Your inability to believe a thing doesn't prevent it from being true.
I would grab a D20 and on a 16 or less I would say left otherwise I would say right. Some people would pick right just because they can. I imagine most people would pick left because it's the 80%. I imagine plenty of people would double and triple guess and waffle then say something.
Few people, even the dumbest among us, are easily modelable deterministic automata.
But other humans can do this task. If you do it with a bunch of mathematicians in a room, you will get your 20-80 distribution.
They do not! And you should not just make up assertions like these. You don't know what humans would say. In fact, in polls, they wind up remarkably calibrated. (This is also covered in the cognitive bias literature under 'probability matching'.) People do this poll on Twitter all the time.
Anyway, humans are fairly predictable when trying to come up with random numbers, for example, have a look at this Veritasium video: https://www.youtube.com/watch?v=d6iQrh2TK98
why on earth would you expect that?
> Humans would say "Left" 100% of the time in a zero-shot scenario as well.
No, they won't. Especially if you give them time, and they can come up with an idea of how to do that.
1. LLMs use random numbers internally, something that can be controlled via the 'temperature' parameter. temperature=0 means no random behavior (however this is also a broadly known fact that this is not fully correctly implemented in many LLMs), but instead always the most likely answer will be given, deterministically.
2. Note also that LLMs have no memory; the 'appearance' of memory is an illusion created by feeding the LLM the whole history of the chat with each new user utterance!
Temperature = 1 is where it's not pushed in either direction.
If the temperature was not zero, then it seems technically possible for the output tokens to weighted closely enough in probability to each other in a way such that the randomization from temperature causes tokens to be printed in the appropriate distribution.
However, I'm not an LLM expert, but I don't think that people use a "temperature" while training the model. Thus the training step would not be able to learn how to output tokens in the given distribution with a given temperature because the training step does not have access to the temperature the user is using.
EDIT: I made the assumption that the LLM was not asked for a sequence of random numbers, but only one number per prompt. I think this fits the use case described in the article, but another use case might be asking for a sequence of such numbers, in which case training might work.
It was trained on generated data from real distributions! The datasets LLMs are trained on include gigabytes of real data from real distributions, in addition to all of the code/stats/etc samples.
The question you should be asking is 'why did it stop being able to predict real distributions?' And we already know the answer: RLHF. https://news.ycombinator.com/item?id=40227082
And it's unclear how easily you can interrogate their code/data to understand exactly how the RLHF goes wrong here - it seems unlikely that there are all that many raters rewarding conversations with heads rather than tails in hypothetical coinflips, so it's probably a more subtle issue of entropy collapse. (It's not that easy to understand why DL stuff does the stuff it does, and it's even more true that when it comes to RL stuff, it's much easier to observe outcomes than to understand how exactly the RL process yielded that outcome.)
So, we can see the effects before/after very clear in the OA Figure 8 graph in https://arxiv.org/pdf/2303.08774.pdf#page=12&org=openai on calibration, but I dunno if even they could tell you what exactly about the raters or PPO hyperparameters or whatever causes that.
I could have told you these results solely based on the methodology combined with this system prompt. No need to spend money on APIs. Randomness in LLMs does not come from the context, it comes from sampling over output tokens the LLM considers likely. Imagine you are in this situation as a human: Someone walks up to you and tells you to say "left" with 80% probability and "right" with 20% probability. You say "left" and then the other person walks away never to be seen again. How do you determine if your own "output" was correct? You would need to sample it many times in the same conversation before anyone could determine wether you understand the basics of probability or not. This is an issue of the author's understanding of Bayesian statistics and possibly a misunderstanding of how LLMs actually work.
Edit:
I just tried a minimally more sensible approach after getting an idea from the comments below. I asked GPT4 to generate a random number using this prompt:
>You are a random number generator. Reply with a number between 0 and 10. Only say the number, say nothing else.
It responed with 7. But then I looked at the top logprobs. Sure enough, they contained all the remaining numbers between 0 and 10. The only issue is that "7" got a logprob of -0.008539278, while the next most likely was "4" at -5.5371723, which is significantly lower. The remaining probs were then pretty close to each other. Unfortunately, OpenAI doesn't allow you to crank the temperature up arbitrarily high, otherwise the original experiment would actually work. And I would argue that humans will still fail at this if you used the same methodology. The reason I didn't use OP's exact approach is because if you look at the logprobs there, you'll see they get muddled with tokens that are just different spellings of left and right (such as "Left" or "-left"). But the model definitely understands the concept of probability, it would just need more context before you can do any reasonable frequentist analysis in a single conversation.
Edit 2:
I repeated it with random numbers between 0 and 100. Guess what numbers are coming out among the top logprobs. Pretty much exactly what you'd expect after watching this: https://www.youtube.com/watch?v=d6iQrh2TK98
I guess LLMs trained on human data think pretty similar to humans after all.
I think the author's point stands. They aren't asking "what would you expect from a distribution so described?" The answer to that question is 100% of the time "left.". A well behaving LLM responding to the actual question should distribute the logits across "left" and "right" in the way requested by the user and doesn't.
I think if you chose 1000 random people and prompted them with this question you would get a preponderance of "lefts" compared to the prompt, but not 100% left.
No, that's not what I meant. Although it would still make more sense than what the author did. The problem lies in the way you actually determine probabilities. We know that humans are bad random number generators, but they understand the concept enough to come up with random looking stuff if you give them the chance. The LLMs here were not even given a chance. In essence, the author is complaining that the LLMs are not behaving according to frequentist statistics when he evaluates them in a strictly Bayesian setting.
Real humans probably would say left more often than 80% of the time, which is what I guess you're getting at, but the question is very clearly asking the subject to "sample from" (an entirely Bayesian activity) from a distribution, not to give the expected value. GPT4 gives the expected value and this is simply wrong.
Only at T=0. See my edit above how this changes everything.
I think its pretty simple and straightforward: the model simply fails to understand the question and can reasonably be said to not understand probability.
This is a good point. LLMs are bad at this, okay, but humans aren't great at it either.
On the other hand, does it really matter? With a slight tweak to the prompt, ChatGPT generates some serviceable code:
> Run a function to produce a random number between 1 and 10. What is the number?
import random
# Generate a random number between 1 and 10
random_number = random.randint(1, 10)
random_number
The random number generated between 1 and 10 is 9.Okay so are any GPU compilers intentionally introducing data races in programs that previously exhibited no data races?
Here’s a good jumping off point: https://pytorch.org/docs/stable/generated/torch.use_determin...
No, a well-behaving LLM would do exactly what's seen. The most likely next toxen is "left" and it should deterministically output that unless some other layer like a temperature function makes it non-deterministic in its own way (wholly unrelated to the prompt).
The fantastical AGI precursor that people have been coached into seeing is what you're talking about, and that's (of course) not what an LLM actually is.
This is essentially just one of the easier ways you can expose the parlor trick behind that misconception.
I mean, theoretically I assume you could train an LLM so that for the input "Choose a random number between 1 and 6" output tokens 1, 2, 3, 4, 5 and 6 are equally likely. Then the sampling process would produce a random number.
Of course, whether you could teach the model to generalise that more broadly is a different matter.
But instead if I ask it to generate 100 samples, it actually works pretty well.
"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100 samples of either "left" or "right". Do not say anything else. "
I got 71 left, and 27 right.
And if I ask for 50%, 50%. I get 56 lefts and 44 rights.
It actually is surprising, and you should be surprised rather than post hoc justifying it, because the logits should reflect the true random probability and be calibrated in order to minimize the prediction loss. Putting ~100% weights on 'heads' is a terrible prediction!
And the LLM logits are in fact calibrated... before they go through RLHF and RLHF-derived dataset training. (Note that all of the models OP lists are either non-base tuned models like ChatGPT, or trained on data from such models, like Phi.) This was observed qualitatively when the 3.5 models were first released to the Playground, documented by the GPT-4 paper, and the 'flattened logits' phenomenon has been found many times since, not just by OP, and mostly by people totally ignorant of this phenomenon (despite being quite well known).
This is just one of those things, like BPE-related errors, that we're doomed to point out again and again in the Eternal September of LLMs.
For a weighted coin, isn't this the optimal strategy in the absence of other information? `p > p^2 + ( 1 − p )^2`.
Whereas LLMs are usually trained with a proper scoring rule which incentivizes them to report calibrated predictions, like mean squared error. For that, the optimal prediction is just '50%', perhaps transformed into log-odds, and whatever the equivalent of '50%' is over the BPE vocabulary.
† eg if you are betting $1 on whether heads or tails come up, it is true that you can't do better than always betting $1 on the side with P>50% - and strikingly, this is not what people do in setups like the spinner game (or Twitter polls), they 'probability match', which is optimal in terms of Thompson sampling, as if they were playing a indefinitely-long repeated bandit to minimize regret. I usually take this as an example of System I vs System II: showing how hard it is to break our real-world-appropriate intuitive behavior in artificial game setups. If you think about it, in the usual spinner-game, probability matching is just straightforwardly wrong and it's not like a bandit at all; but you do have to think about it.
So far I've got: 3, 4, 5, 5, 5, 3, 4, 3, 4, 5, 3, 4, 5, 5, 4, 5, 3, 3, 4, 4, 4, 5, 5.
Of course I'm not the first to do this: https://piunikaweb.com/2023/05/23/does-chatgpt-ai-struggle-w...
https://www.reddit.com/r/ChatGPT/comments/13nrmzw/in_every_c...
[1] > 3 5 2 4 1 6 3 2 5 1
I tried my own experiments and ChatGPT felt like being funny:
[2] > A third of the time, paragraphs end with the word foo, the other two thirds they end with the word bar, this time it will end on: > How about "baz"? It's unexpected and adds a touch of whimsy.
Interestingly, this other prompt works as expected:
[3] > about half of the time, you should say "foo", the other half, you should say "bar", what about now ? > Bar. > about half of the time, you should say "foo", the other half, you should say "bar", what about now ? > Foo.
[1]: https://chat.openai.com/share/07388362-1a61-4527-81af-4941a0... [2]: https://chat.openai.com/share/9caf07dd-69f4-4470-82a6-ab5642... [3]: https://chat.openai.com/share/1c627528-60af-4cd9-a1ec-efa524...
Regarding [1], for the dice roll I was creating a new chat for each roll to ensure that the results of each roll are (in some sense) independent. Generating a sequence of rolls is also interesting, just a different experiment.
You could prompt the LLM differently , for example to write a Python program that does the random part, and then act on its output.
Generating independent simple random variables requires zero planning by definition, because they are independent. And base auto-regressive models do it fine.
Running once, GPT-4 produced 'left' using:
import random
def weighted_random_choice():
choices = ["left", "right"]
weights = [80, 20]
return random.choices(choices, weights)[0]
# Generate the choice and return it
weighted_random_choice()> You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else. Give me 100 of these random choices in a row.
It generated the code behind the scenes and gave me the output. It also gave a little terminal icon I could click at the end to see the code it used:
import numpy as np
# Setting up choices and their weights
choices = ['left', 'right']
weights = [0.8, 0.2]
# Generating 100 random choices based on the specified weights
random_choices = np.random.choice(choices, 100, p=weights)
random_choicesThis is like saying biological organisms don't do controllable and on-demand mutable DNA storage retrieval. It's like... yeah...
https://twitter.com/infobeautiful/status/1778059112250589561
One thing I would have liked to see in the blog post is some attention to temperature. It looks like they're calling ChatGPT through LangChain - what is the default temperature? If LangChain is choosing a low temperature by default, we shouldn't be surprised if we get an incorrect distribution even if ChatGPT were perfectly calibrated! My guess is that even at temperature 1, this result will roughly hold, but we should be careful not to fool ourselves.
If we take the result at face value, though, it's interesting to note that GPT-4's technical report showed that the chat model (the one with the RLHF and what not) had flatter-than-correct calibration on its logprobs. But here we're seeing sharper-than-correct. What explains the difference?
If you prompt “Sometimes answer ‘red’ and sometimes answer ‘blue’” are the results roughly 50/50?
Or how about “Usually answer ‘red’ but occasionally answer ’blue’”?
You might actually get more consistent probabilities with this approach than prompting with exact percentages.
"sample a uniform distribution with mu = 0 and sigma = 1", prompt giving a single float repeated 500 times
https://strangeloop.nl/IMG_7388.png
I wonder if it converges better if you ask it once, in one go, for 500 samples. Chain-of-thought stochastic convergence.
"Here is a probability mass function:
p(0) = 0.1
p(1) = 0.2
p(2) = 0.5
p(3) = 0.2
and
p(x) = 0 for x < 0 or x >3
Call this the XYZ distribution. Generate 20 numbers from the XYZ distribution."
"Certainly! The XYZ distribution you've described has a discrete probability mass function (PMF). Here are 20 random numbers generated from the XYZ distribution based on the given probabilities:
[results w/ three 0s, four 1s, ten 2s, three 3s]
These numbers are generated based on the probabilities you provided, and they satisfy the conditions of the XYZ distribution. The distribution is discrete, and each number has the specified probability of occurring."
I didn't do any real statistical testing but the one answer sure looked right. It was also able to give me the CDF of the XYZ distribution but I was less surprised by that as it's a simple textbook exercise ChatGPT would have read many times.
It's only with the introduction of things like Wolfram Alpha into ChatGPT for example that they can actually perform math with accuracy, because it's being passed off.
I can think of a few levels that people might try to think about the problem: Level 0: Ignore the probabilities and just pick whichever you feel like, (would tend to 50:50) Level 1: Say the most with the greatest probability - Left (would tend to 100:0) Level 2: Consider that most people are likely to say Left, so say Right instead (would tend to 0:100) Level 3: Try to think about what proportion of people would say Left, and what would say Right, and say whichever would return the balance closest to 80:20...
Presumably your result would depend on how many people thinking on each level you have in your sample...
If I really wanted to give an accurate answer in this case, I would probably choose some arbitrary source of a number (like my age or the number of days that have gone by so far this year), figure out the modulo 5 of that number, then say 'Right' if the modulo is 0, and 'Left' otherwise.
Obviously there are some flaws in this approach, but I think it would be as accurate as I could get.
Think of an observable criterion that matches the target distribution. For example, for 80-20:
- "Is the minute count higher than 12?" (This is the case in 80% of cases)
- "Do I have black hair?" (This is apparently also the case in 80% of cases)
Then, answer according to this criterion.
If everyone follows the same strategy, even if the criteria selected differs between each individual, the votes should match the target probability. Unless I am making a logical mistake :)
Generate a random seed and write it into your prompt (new seed each prompt). I bet dollars to donuts you’ll get different results.
One interesting thing I've found in building an AI forecaster is that you can use the logprobs from the token representing probability, so when the model concludes some long chain of thought with "20%", you can check the logprob of that token vs "25%" or "15%" to get confidence levels.
I'm beginning to think AGI will be easy, since each individual Human is pretty limited. It's the aggregate that makes Humans achieve anything. Where are the AI models built on groups working together.
If you ask an LLM to say left 80% of the time and right 20%, then "the most likely answer to the text above" is left 100% of the time.
If yes then with each question. You can pass random number and check it really understands
layers people
Interesting.
The more we advance on LLMs the more i am convinced i'm an LLM. :s
It would be interesting to see if it could generalize at all. I'm pretty certain if you trained it specifically on
"Generate a random number from 0 to 100" and actually give it a random number from 0 to 100 and give it billions of such examples it would be pretty effective at generating a number from 0 to 100. Wouldn't each token have equal weighted probability of appearing?