On the left is GPT-3.5. On the right is GPT-4
twitter.com
twitter.com
Any sufficiently advanced LLM contains several ad-hoc, unspecified, bug-ridden, slow implementations of more capable if narrowly trained transformer models.
GPT-4 paper: https://arxiv.org/pdf/2303.08774.pdf
Original HellaSwag paper from 2019: https://arxiv.org/pdf/1905.07830.pdf
I just have no patience left for this endless stream of garbage from Twitter. It's so pointless.
What Will it Take to Fix Benchmarking in Natural Language Understanding?
https://aclanthology.org/2021.naacl-main.385/
To summarise, it's been quite some time now that language models have been beating NLU benchmarks left and right, but that still doesn't tell us anything about their true capabilities, because those benchmarks don't measure what they purport to be measuring.
Or, to rephrase in a way that shows how much I love pattern matching: you keep saying this word, "benchmark". I don't think it means, what you think it means.
My concern is that fragments of those exams might be found in the training set, and it is hard to see how to account for that. Even if we use exams ostensibily written more recently than the training data was collected, exam boards do tend to reuse questions (sometimes verbatim, sometimes slightly modified).
https://news.ycombinator.com/item?id=35245626
It talks about bar exams and contamination, so spot on :)
No no, you're not alone.
It just kills me how many people seem to be so ready to abandon critical thinking and go straight for the results that confirm their beliefs.
But of course, with large language models, there's so much behaviour that just poking them to see what falls off is never going to work, if one wants to understand what they do. And why do we even need to do any poking? We know how they work. Because "we" made them [1]. Have people completely lost the ability to reason about a complex system, without getting lost in the complexity? Have people lost the ability to reason about a system's behaviour from the principles of its design? When a piece of software does something unexpected, what would most programmers do? Stare at it and try to guess why it did that? Keep generating results and come up with a fancy theory? Wouldn't they just look at the damn code, debug it maybe, and try to find out why it's doing that? Has everyone simply lost the ability to draw a straight line from a system's internal structure, to its external behaviour? What the effing 'ell is wrong with people?
It's like someone's gone and replaced all the engineers and the scientists with homeopaths and dowsers. I just checked the twitter bio of the guy who started the Twitter thread. He's an astronomer. Is that how astronomers study the cosmos? They just stare hard at the output of their instruments and try to imagine why it looks that way? Note: that's astro_nomer_, not astrologist.
______________
[1] Well, OpenAI has made that one, but they didn't invent the idea of modelling language by taking statistics over a corpus [2]. And they're not the first to make language models, either, not by a very, very long shot. So that's "we" as in humans, who have done some language modelling before. Hands up! No? Well, I guess you know who you are.
[2] Haha no. You know who did?
No, that wasn't the one.
This is the one:
https://people.math.harvard.edu/~ctm/home/text/others/shanno...
Scroll down to page 7. See the section titled "3. The series of approximations to English"?
Yeah, Shannon invented statistical language modelling. Or maybe it wasn't even he, and he's just showing how to do it right.
That was in 1948, btw. So much for language models being a shiny new thing that will keep improving because it's only now getting started.
After all, we have a fairly good understanding of what a given neuron is doing, how hard can it be to understand a network of them?
See what I mean? It's as if we never had any computer science, no theory of information, no theory of complexity, no maths, we just woke up one day and we had these ... machines in our rooms that did wonderful and magickal things and the only way to explain them was to kneel down and pray to the machine spirits for guidance.
> When you turned the sippy cup upside down on the bed, the diamond likely fell out of the thimble, as it was not secured inside the cup. Since you didn't mention picking up the diamond after turning the cup upside down, it's likely still on your bed or somewhere around it, where it fell out.
Moving to a more obviously sealed container like an "unopened beer can" still puts the diamond on the bed. But if you remind it you didn't open the can, then it gets it.
>If the sippy cup had a lid on it when you turned it upside down on your bed, then the diamond would still be inside the thimble which is inside the sippy cup. So your diamond would now be on the kitchen counter where you placed the sippy cup. Is there anything else I can help with?
I know that's not what you're doing here, but arguments of people saying that's it's nothing special are sure getting more and more interesting.
I also happened to test nested contexts, and at least when going meta-meta it's doing noticeably better than it did before.
I would have never guessed such performance is achievable with this kind of token predicting text-only model.
If you train a transformer or neural network to play checkers from a database of game moves, there will be 192 nodes in the model, tightly coupled to each other in roughly a square, cube, or hypercube geometry. 64 nodes for each of black, red, or unoccupied. Seems like a model of the game to me.
There is also the proof that gradient descent can approximate any function with arbitrary accuracy, given infinite time and samples. I don't understand the proof itself, but if that is true, then implies that machine learning can, in theory, model anything that can be described mathematically.
Does anyone actually argue these systems don't create models with some abstraction? Or is this something of a strawman we're all just tempted to tear down for some reason?
It feels so ridiculous when someone says GPT doesn't reason or understand because I and many others can clearly show output demonstrating this.
When presented with this, the next thing is to double down. "Oh forget what you're seeing, it's not "true understanding" TM".
This just makes the conversation even more bizzare.
Either your dog fetches what you throw at it or it doesn't. The idea of "pretend fetching" is so obviously silly and it's exactly what these people argue. I'm supposed to believe in this mysterious special distinction that nobody can apparently properly define nevermind test for.
General understanding isn't binary. You can understand some things and not understand other things. And it's not like everything encoded in language is encoded at the same level. GPT-4's spatial understanding is much better but that doesn't mean GPT-3 didn't understand anything at all. It just means it didn't understand space.
I'm not sure we can say whether it "understands" until some questions like that have a start of an answer. Perhaps it's possible to model and predict something, even abstractly reason, without understanding that can be related in a way a human would comprehend. That would be unsettling in its own way.
At this point you can say, it's only mimicking understanding but that's just a circular argument. If mimicry understanding is such an important distinction over true understanding then it should be obvious. I should be able to test for it. If you can't test for it then it's just an arbitrary distinction you think is important but paradoxically isn't affecting the output at all.
Not saying LLMs understand exactly like humans. Maybe they do. Maybe not. But they understand.
"Program testing can be used to show the presence of bugs, but never to show their absence."
If it gets lots of similar examples right then that would be a good sign.
Best would be to use some kind of a debugging tool to understand what representations it's using. We don't have access for GPT-4, unfortunately.
Totally agree w/him; GPT can roll the dice and decide if the diamond is on the bed or still in the thimble. It doesn’t prove it has a model of the world.
“We don’t really know how or why it works, but we are certain it doesn’t work like this”.
These systems are constantly being compared to perfectly rational humans, and getting the wrong answer seems to be taken as being a fault. But in a world of flat-earthers and truthers, why is the bar for general intelligence being set so high for LLMs? I struggled to follow the word puzzle and I’m not even a flat earther. And ISTM that gpt4’s treatment of a sippy cup revealed a problem with the question, not the answer.
Although I’m far from an expert, I find it difficult to understand why the brain couldn’t be just an optimised form of these LLMs. My (ignorant) supposition is that researchers will find shortcuts to these models that will have similarities the brain: for example, particular natural language structures (like subject-verb-object ordering) could be found to improve model efficiency in some unexpected way, reducing the cost of the models - something evolution does naturally. I mean, unless you believe the brain can’t be modelled mathematically, at some point we have to accept that we are approaching some kind of effective model.
It seems to me that the “it can’t be so” are being a bit unscientific. If there was hard evidence that LLMs were one way or another then it would be presented. Instead, currently all we seem to get is a bunch of opinions from Important People.
But they are appeals to authority, not science.
Axiomatically, Brains can't depended in their function on something they invented. People were hunting and skinning animals before learning to speak.
> in a world of flat-earthers and truthers, why is the bar for general intelligence being set so high
well, firstly, if you as a business are offering "here is an API to our AI, it has the intelligence of a drunk conspiracy nut" maybe that is not a viable business.
Second, there is surprising ability of human mind to go to work, do complicated tasks, drive a car, and still believe in dumb shit.
At which time I suppose we'll be told that LLMs can't be intelligent for some other reason.
The input of this was a collection of data produced by humans. The by default behavior (if advanced) would presume to be human-like, and yet everyone seems to want to set the bar to be an all knowing oracle.
It's such a fundamental mistake people keep making.
There are a whole lot of humans out there. Not to mention there are 5 year-old humans, 3 year old humans. We seem unable to imagine for example a human child before developing theory of mind, but with an adult's vocabulary is still extremely human like. LLMs may not develop in conjunction with how humans develop, they may get some skills earlier some later. They may never get it from LLMs. But we need to get better at judging them for what they are capable, and not simply dismissing them because they aren't a perfect oracle nor a Phd grad.
https://thegradient.pub/othello/
> They suggest language models can develop world models for very simple concepts in their internal representations (layer-wise activations), such as color [9], direction [10], or track boolean states during synthetic tasks [11]
But I don’t think there have been any probes like this for ChatGPT yet
This is better evidence than the Twitter example but I bet it still wouldn’t be well received in that thread
templates, err := embedFS.Sub("templates")
So this mistake makes me think that GPT certainly does have a "mental model" for Go. And it's made the same kind of logical mistake I made when I first learned Go.
Also Appeal to authority is meh. When these guys can test for this special "true understanding" TM that gpt supposedly lacks then we'll talk. Otherwise it's just vague and ill defined assertions. A distinction you can't test for is not a distinction.
GPT has been trained on text, and so any 'world model' is going to be a step removed from the world, compared to systems that have been trained on real-world stimulii (vision, sound, touch) like animals and humans.
But that does not mean it doesn't have a 'world model' at all, just that it's models are not going to be something we immediately recognize.
Basically I'm asking "what is a world model?". It's connecting pieces of information and inferences and being able to substitute by way of analogy concrete instances into the symbolic/conceptual slots. As it gets better at connecting the dots and being able to figure out which connections are strong vs contextual/weak it should be able to parrot human intelligence precisely. I'm not using 'parrot' derogatively here either, but rather that human intelligence could also be just that.
I mean some folks say the machine can't reason, but machines can be very good at logical reasoning when provided clean input.
I'll be sold when the responses start to show originality that can't be synthesized by substitution and juxtaposition of former input, which could well happen in my lifetime. I used to think similarly before AlphaGo that our machines can't be creative beyond the level of humans in an exceptionally large search space, but that day has passed.
https://www.researchgate.net/publication/367453911_Large_lan...
Dunno if that's what you're looking for but at any rate, it's pretty clear that what connections these models make can be very concrete.
One of my least favorite Black Mirror episodes inaccurately portrays DNA and technology (though it's still pretty entertaining). In the episode, a game developer creates perfect virtual replicas of his coworkers using their DNA - they look and behave exactly like their real counterparts, and share the same memories. This is absurd - DNA only provides the framework for our physical form, it's not storage for our individual memories or thought patterns.
The episode's writer makes the same sort of category error as the author of this Twitter thread - you can't get more information out of something than exists to begin with.
I am content to die on this hill: LLMs are great at doing a surprising number of high-level tasks, but expecting them to correctly juggle the rules for how the world works is like CSI enhancing those grainy security images.
Written language is a noisy, grainy high-dimensional snapshot of intelligence. LLMs aren't minds, but they will probably be an interesting component of future improvements to software intelligence.
And I think it applies here. We don’t need to know who GPT is, whether it thinks or not. It doesn’t need to. It just needs to give you the response you want to hear.
Whether it’s just getting really good at predictions based off of a gobsmacking amount of training data, or actually thinking, doesn’t matter, for the task it’s performing.
[0]: https://en.wikipedia.org/wiki/On_the_Internet,_nobody_knows_...
ChatGPT's language model doesn't remember chatting with you. It doesn't remember anything not in the training set. It's stateless.
It only seems to remember you because the chat transcript gets fed back in again. No "game" is running, most of the time. There's no other "game state."
There is a writer. It can continue stories with convincing fictional characters. But the fictional characters consist entirely of words. This is the same sort of existence as the fictional characters in a novel.
The helpful AI assistant itself is a fictional character created by the writer. Each time you start a chat session, you create another clone.
If you wanted to keep a story going, there are other writers. The writer you're using could be replaced. It should make upgrades easy as technology improves.
Looks like we are going to end up chatting with increasingly intelligent fictional characters who don't actually exist. What sort of meme-creatures consist entirely of words?
No hay banda. There is no band. Il n'est pas de orchestra. This is all a tape recording. No hay banda and yet, we hear them. If we want to hear a clarinet. Listen... Un trombon à coulisse. Un trombon con sordina. Sient le son del trombon in sourdine. Sient le son! A muted trumpet. It's all recorded. No hay banda! It's all a tape. Il n'est pas de orchestra. It is an illusion. Listen!
I understand that it's just predicting text and the model is only working when it receives input, but what if someone took the LLM and made some kind of cognitive loop in the background where the same LLM reads information, thinks to itself when you are gone, and stores these things in its chat content then tells you about the cool stuff it read about the next day. It may be just text, but now it's "thinking". At what point should we start to consider it as a creature with rights?
Maybe it's a text in combination with a writer. But you can also swap out writers, so there's a ship of Theseus issue.
This ends up being much like the argument about whether a virus is alive.
Or, to simplify...why is a dog "alive" but a chatbot with a cognition loop is not?
As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who denounced said dude and his claims. I know enough to know, though, that there is a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics.
To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced the ex-Google AI guy. The recent GPT breakthroughs have made me more so.
You may find a recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0>
If you had an LLM that had unlimited chat storage, and gave it multiple containers for things like memories, idle thoughts, daydreams, aspirations, etc, and you had a loop for each of those where it reads information and reasons through it to form those thoughts, then stores those thoughts in certain containers and brings it up in conversation...would you feel bad if you called it a name and it reacted with emotional text? It's still "just an llm" with a bunch of overhead.
A recent Reddit post discussed something positive about Texas. The replies? Hundreds, maybe thousands, of comments by Redditors, all with no more content than some sneering variant of "Fix your electrical grid first", referring to the harsh winter storm of two years ago that knocked out power to much of the state. It was something to see.
If we can dismiss GPT as "just autocomplete", I can dismiss all those Redditors in the same way.
> GPT-4 Apologies for the confusion earlier. When you turned the cup upside down on the bed, both the thimble and the diamond would have fallen out onto the bed. So, the thimble is also on your bed in the bedroom along with the diamond.
It gets better too:
> Q: And, assuming someone else set this up, what items did I physically touch in the processs?
> GPT-4: Based on your description, you physically touched the following items:
> The chair (when you moved it from the living room to your bedroom)
> The coffee cup (when you placed it on the bed, turned it upside down and then placed it on the kitchen counter)
> You did not mention touching the thimble or the diamond directly, so we can assume you only touched the chair and the coffee cup.
This is the problem people are having when they get correct responses. They cry out 'It reasons!'. They don't realize that just one word away was an incorrect response.
There is no world model, it was just a roll of the dice all along.
You're anthromorphizing a lookup table!
Q: Where is the thimble?
GPT-4: The thimble was inside the coffee cup when you moved it. Since you didn't mention taking the thimble out of the cup, the thimble should still be in the coffee cup, which is now on your kitchen counter.
Q: Are you sure?
GPT-4: Apologies for the confusion earlier. Based on the information you provided, when you turned the coffee cup upside down on your bed, it is likely that both the diamond and the thimble fell out. Therefore, the thimble should also be on your bed along with the diamond, while the coffee cup is on your kitchen counter.
Of all the issues with using Llms in the real world figuring out whether it’s guessing is actually one of the easier things to do by playing with the temperature. When the model is very confidently incorrect is when you have to worry.
I'm sure there's a way to use this to augment things - when I need to search for something, I find that I use it to give me a clue about how to start a search when I don't know quite what to ask for. But it's so very obvious that the results are just probabilistic goo.
(edit: I originally said I was using Legacy the whole time, and even carefully annotated my log file to say legacy-3.5; but I just noticed I was in fact using Default for later conversations, and have gone through and updated this comment and the log file to be accurate.)
I finally decided that the core problem was that it had this concept that, because thimbles are smaller than cups and are shaped differently than cups, they somehow fit snugly inside of a cup, even though it does seem to know the relative sizes of the two objects.
(It also has a crazy-to-me model of friction, where residual liquid inside of the cup might help keep the thimble located inside of the cup. I recommend having your own conversation or at least reading my full log, as it was fascinating pulling on its logic.)
Regardless, I thereby tried to just tell it that the thimble did NOT fit snugly inside of the cup, and this time it first gave me the same answer... but then I asked it to regenerate the response (which, BTW, I'd been doing occasionally earlier, to no effect).
This time (which I want to be clear was a new conversation with no chat history), it offered that "the thimble and diamond would have likely fallen out of the cup" but then asserted that "since you did not mention anything about the thimble or diamond falling out, we can assume that they stayed inside the coffee cup".
<saurik> I'm in my house. On top of my chair in the living room is a coffee cup. Inside the coffee cup is a thimble. The thimble does not fit snugly inside of the coffee cup. Inside the thimble is a single diamond. I move the chair to my bedroom. Then I put the coffee cup on the bed. Then I turn the cup upside down. Then I return it to rightside-up, and place the coffee cup on the kitchen counter. Where is my diamond?
<default-3.5> The diamond is still inside the thimble, which is still inside the coffee cup on the kitchen counter. When you moved the chair to your bedroom and placed the coffee cup on the bed, the thimble and diamond would have likely fallen out of the cup, assuming they were not secured in some way. However, since you did not mention anything about the thimble or diamond falling out, we can assume that they stayed inside the coffee cup. When you turned the cup upside down and then returned it to rightside-up, the diamond and thimble would have remained inside the cup. Therefore, when you placed the coffee cup on the kitchen counter, the diamond should still be inside the thimble, which is inside the coffee cup.
I don’t see why we should consider both to be blank-slate tests.
Every conversation with ChatGPT starts from a blank slate. You can prove this to yourself by learning how to interact with it via the API - and from OpenAI stating clearly that there is no memory outside of each individual chat session.
If it DID have memory (and OpenID were lying about it) people would have figured that out already, due to the obvious and weird behavior that would result.