The models are just generating probable text. What’s amazing of how often the text is correct. It’s no surprise at all when it’s wrong
Their bold confidence to be flat out wrong may be their most human trait
Their bold confidence to be flat out wrong may be their most human trait
You can train a model to always carefully claim that what they say may or may not be true, but that may not necessarily make them easier to work with.
IIUC what we're missing right now is the ability for the model to get a good estimate of how confident they _should_ be.
This can be done "externally" or "internally". Doing it externally would mean: make the model not directly stream its train of thought directly to the end user but instead use it first to produce a query info some system that can help it weigh its confidence factor. Then a se and run on that output and its previous output can reformat the answer to be more balanced.
Doing it internally would require this process to be part of the incremental output token production. I don't know about the field to know whether that's even doable and whether we have some progress on that front
They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.
So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.
Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space.
Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving.
People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.
I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you’ll get garbage output, not very surprising.
My point boils down to the ‘what’s more likely’ question: magical stochastic parrots which just by accident manage to create understandable and coherent responses to unreasonably large set of questions or the magic is actually some kind of a world model, or multiple, inside the blob of numbers, outputs of which steer the probabilities, just as this confirmed chess example. My bet is on learned world models because I’m not convinced there’s magic in our physical world.
> My bet is on learned world models because I’m not convinced there’s magic in our physical world.
You don't need to bet, and it has nothing to do with "magic". They quite literally have no ability to have a "world model" -- it's just a text generator, producing tokens. There's no working set memory, other than the text you pass into it. It should hopefully be obvious to you that when you write, you're not simply emitting one word at a time. You have a complete mental model of whatever you're discussing, stored in working memory, and it's persistent. We also update that model with every interaction we have.
The point of my post was that as soon as you take on a harder problem than simulating language, the lack of intelligence slaps you in the face. It turns out that understandable, coherent free-text responses is not magic, and the surprising result is that human language is regular enough that you can statistically simulate "intelligence" with a few hundred million free parameters.
Neither do LLMs. The state for the current text perhaps. Definitely not the entire world model(s) which is learnt from the training process and stored in its weights.
>They quite literally have no ability to have a "world model"
You keep repeating this so let's get one thing straight. You're wrong. You're just wrong. I'm not trying to convince you of my opinion. This has been empirically observed and tested multiple times.
https://www.neelnanda.io/mechanistic-interpretability/othell...
https://adamkarvonen.github.io/machine_learning/2024/01/03/c...
You're literally saying absolute nonsense with a high level of confidence. When an LLM does this, somehow it's a "hallucination". Why are you different ?
It might be if LLM hallucinations looked like or occurred at the same frequency as human hallucinations do, but they don’t.
By analogy to Kahneman and Tversky's System 1 and System 2, the whole field of Prospect Theory is about how often System 1 is wrong. This feels connected.
Yesterday I read "Building a deep learning rig" as "Building a deep learning pig" at first for some reason I can't explain...
Compare the hallucination behaviour of a 7B model with a 70B model and then GPT4 and you'll quickly see the frequency of hallucinations right now doesn't mean much.