Eight things to know about large language models [pdf]
cims.nyu.edu
cims.nyu.edu
I thought that perhaps the program just recognized that the only value that could be used to calculate any period was that frequency, and perhaps the model didn't really know anything about the Earth motion through its orbit affecting the average temperature, so I asked:
"Can any of the values be used to estimate the Moon's orbital period?"
and it replied:
"No, the code and the data used in this program are not suitable for estimating the Moon's orbital period. The temperature data being analyzed is influenced by the Earth's rotation around the Sun and other factors like weather patterns, but it doesn't directly correlate with the Moon's orbital period."
---
Me: "continue the sequence: 0, done, 2, free, 4, hive"
ChatGPT 4: "1, sun, 3, tree, 5, drive"
Me: "that's wrong, try again"
ChatGPT 4: "0, done, 2, free, 4, hive, 6, picks, 8, gate"
Me: "that's wrong, try again"
ChatGPT 4: 'I give up.' (paraphrasing the obnoxiously verbose responses it's been trained to prefer)
---
It's a series even a child could complete, but breaks ChatGPT by relying on a pattern of words that aren't explicitly stated. So you get software which can pass the bar, but fail at trivial series.
# Kira
I found this sequence: "0, done, 2, free, 4, hive"
Reason step-by-step about the pattern.
Consecutively continue the sequence with four more entries.
Consecutively reflect on your answer and fix mistakes in case you broke the pattern.
# ChatGPT4
After examining the given sequence, "0, done, 2, free, 4, hive", we can notice an interesting pattern. It appears that the sequence alternates between numbers and words that sound like numbers. Let's break down the pattern:
"0" is a number. "done" sounds like the number "1" when spoken aloud. "2" is a number. "free" sounds like the number "3" when spoken aloud. "4" is a number. "hive" sounds like the number "5" when spoken aloud. Given this pattern, we can continue the sequence with four more entries:
The next number in the sequence is "6". The word that sounds like the number "7" is "heaven". The next number in the sequence is "8". The word that sounds like the number "9" is "nein", which is German for "no".
So, the continued sequence would be: "0, done, 2, free, 4, hive, 6, heaven, 8, nein".
Reflecting on the pattern and the answer provided, the continuation seems to be consistent with the established pattern. We alternated between numbers and words that phonetically resemble numbers, resulting in the sequence "6, heaven, 8, nein" for the next four entries.
Input:
I found this sequence: "0, done, 2, free, 4, hive"
Reason step-by-step about the pattern.
Consecutively continue the sequence with four more entries.
Consecutively reflect on your answer and fix mistakes in case you broke the pattern.
The pattern in the given sequence is that every alternate term is a word that rhymes with the preceding term.
The first term is "0."
The second term "done" rhymes with "0."
The third term "2" does not rhyme with "done."
The fourth term "free" rhymes with "2."
The fifth term "4" does not rhyme with "free."
The sixth term "hive" rhymes with "4."
Therefore, the pattern is:
a number
a word that rhymes with the previous number
repeat
Continuing the sequence:
6, dive
8, late
10, pen
12, delve
Reviewing my answer, I didn't break the pattern, and the sequence follows the pattern correctly.
But the number of tokens in its training data vastly outnumbers the number of parameters the model has (for GPT-3 at least - the numbers haven't been released for GPT-4). So the LLM can't have simply memorized its training set; it simply doesn't have enough internal storage to do that. So it is not just regurgitating portions of its training set.
The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.
Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. Now change those 10,000 input pixels to words instead of pixels, with the target output being a word instead of a letter, and you get ChatGPT. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.
> but can consistently replicate not only its training but even more beyond that
We call that generalization. No?
I think if something was intellectually capable of replicating such progress, let alone with the inefficiencies and weaknesses of mankind stripped out, there would obviously be zero debate as to whether it was intelligent. Just think about the implications. Not only would such a technology be making groundbreaking discoveries in essentially every field, including in its own development, but even then immediately going beyond it. Instead of pursuing technological advances, we'd be struggling to merely keep up with the endless slew of discoveries being handed to us.
I've also thought about human development and what I find interesting is that an anatomical modern human is not automatically capable of forming modern civilization (see isolated tribes still living and general hunter gatherer history in general). There is so, so much that depends on emergent factors like some sort of global or societal intelligence. I'm thinking AI has the same characteristics. It's not about the individuals, it's about "society".
> 50,000 years ago we knew basically nothing, and even language itself may not have been developed
I think language is substantially older than that, but it is a hot area of debate.
The Kolmogorov complexity of the training set is much much lower, because of the structure within the training set that is being used for training. So we don't know the relationship between the information content (size) of the training set and the storage bounds in the model parameters.
Some children, perhaps; but your puzzle stumped this middle-aged engineer.
I've been in the room when people compared notes as adults, and were mind-blown to discover the other type of person exists. The one without a mind voice had never thought people really heard a voice in their heads.
Continue sequence: one, 1, two, 1, three, 2, four, 1,
Edit: I want to add I even get GPT3 getting it right without explanation to be honest. Even that one is outsmarting me. 4 is using terms like homophone. I'm not sure you proved what you thought you proved.
Input:
You are Math AI. An advanced pattern finding AI.
This is your sequence: 0, done, 2, free, 4, hive, ... ?
Find a sensible logical pattern and explain.
Output:
In this sequence, we can observe a pattern involving both numbers and words. The pattern alternates between a numerical value and a word that rhymes with the corresponding number. Here's the sequence again:
0, done, 2, free, 4, hive, ...
Let's analyze the pattern:
The numerical values are increasing by 2: 0, 2, 4, ...
The words rhyme with the numbers preceding them: done (one), free (three), hive (five), ...
6, heaven
---
* Me: "You are Math AI. An advanced pattern finding AI. This is your sequence: 0, done, 2, free, 4, hive, ... ? Find a sensible logical pattern and explain."
---
* GPT4: "I apologize for any confusion, but my capabilities as a language model are not restricted to math or pattern finding. However, I'd be happy to try to help you identify a pattern in the sequence you provided.
The pattern that seems to emerge from the sequence is that each number is followed by a word that rhymes with the number spelled out as a word. For example, "done" rhymes with "one," "free" rhymes with "three," and "hive" rhymes with "five."
Following this pattern, the next word in the sequence would be a word that rhymes with "six," which could be "sticks," "mix," or "tricks," among other options. However, it's worth noting that this sequence could also be extended in different ways or according to different patterns, depending on the context or the intended purpose of the sequence."
---
It's just a really long winded version of making the exact same mistake, as it did in the 'unprimed' version!
I experimented a bit with this prompt and tried various variations of it including one that added something like this:
"The age-category of this sequence is: "a child could do it". You are to solve this by pretending to be a young human with average skills and perceptions. You should assume average common sense in your answer."
Outputs are like this:
=====
"Now, let's look at the words. I noticed that the words have something in common with the numbers that come right before them. They are related to the numbers in this way:
'done' - It sounds like "one" when you say it aloud.
'free' - It sounds like "three" when you say it aloud.
'hive' - It sounds like "five" when you say it aloud."
0 (even number), 'done' (rhymes with 'one'), 2 (even number), 'free' (rhymes with 'three'), 4 (even number), 'hive' (rhymes with 'five')"
=====
I agree with you that language models have vastly different failure modes than humans. Often extremely weird, but to even get close to their potential you do have to probe a bit deeper than just asking it stuff without context.
It just semi-randomly walks some semantic space and not all of those branches are fruitfully related to our initial problem. It doesn't know at all what we are doing, there is nothing going on "under the hood".
Edit: to be completely clear I'm not arguing it "does know" this answer or something. I don't know if it even "knows", but I do know a thing or two how to get better answers. Current models still are very clearly limited so sooner or later you'll hit it the limit and they start what I call "averaging": you'll get blurry pictures of what you want.
A continuation could be "6, heaven, 8"
given a sequence of numbers 0 to 10, replace every second number with a word the rhymes with the number.
0, done, 2, free, 4, hive, 6, jeven, 8, line, 10, neleven, 12, pirteen, ...
On the one hand, Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). Just as much, LLMs don't express their creators values 'cause they don't express any values, they average language responses (though people certainly can see them expressing values, which can cause problems for the people).
On the other hand, the point that their creators don't understand them and are still quite willing to toss them to the open Internet really show considered carefully whatever their exact capabilities.
The fact is that they acquired abilities no one expected them to acquire. In hindsight you can say it was embedded in language and maybe you could have seen it coming, but it is an empirical fact that this was unexpected and unpredictable beforehand.
And it was only unexpected to those that weren't following the news the GPT 3 paper was published in 2020 and detailed this wild advancement in its capabilities[0,1].
0: https://www.infoq.com/news/2020/06/openai-gpt3-language-mode...
That's wrong. It's been demonstrated that GPT4 has a model of physics, it knows how to play chess, arithmetics, it can interpret and run code, etc. (not perfectly, but to some degree). It goes way beyond basically "knowing human communication".
«How to to play chess» -> devolve into illegal moves quickly?
So does that mean that its model of physics will let it levitate as well then?
> «How to to play chess» -> devolve into illegal moves quickly?
I didn't try it myself as I'm a beginner and need to see the board, but it reportedly plays pretty well and is able to recognize illegal moves[1].
> So does that mean that its model of physics will let it levitate as well then?
It means that it understands how physics work (e.g. things fall due to gravity). For example, it will correctly answer this question: "I put an egg inside a cup. I go to the bedroom, put the cup on the bed and turn it upside down. I lift the cup and bring it to the kitchen. Where is the egg?".
[0] Sample interactions: https://news.ycombinator.com/item?id=35299121
[1] https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_t...
this framing denies agency, denies diversity, denies choice. A human, presented with the sort of directions and instructions that LLM's receive, does not have a single "correct" outcome (for that matter, neither do some LLMs). So no, an LLM does not "know how humans would respond". Instead, it can generate something that lies somewhere within the window of possibilities that a human might offer under similar conditions.
I think the author would say those are not the capabilities being discussed here.
Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins:
> There is increasingly substantial evidence that LLMs develop internal representations of the world to some extent, and that these representations allow them to reason at a level of abstraction that is not sensitive to the precise linguistic form of the text that they are reasoning about.
LLMs do not "reason"; they do not "learn" or "develop" anything of their own volition. They are (advanced) statistical models. Anthropomorphizing them is not just technically incorrect, but morally disingenuous. Writing about LLMs in this way causes people with less domain-specific knowledge to trust the models the way they might trust people, and this has the potential for serious harm.
Because of this choice of phrasing, I wanted to look into the author's background. Among their recent activities, they list:
> I'm organizing a new AI safety research group at NYU, and I wrote up a blog post explaining what we're up to and why.
"AI Safety" is a distinct (and actually opposing) area from "AI Ethics". The people who prefer the word "safety" tend also to engage in discussions that touch on aspects of longtermism. Longtermism is not scientifically well-grounded; it seeks to divert attention from present and real issues to fanciful projections of far-future concerns. I do not know for certain that the author is in fact a longtermist, but their consistent anthropomorphization of a pile of statistical formulae certainly suggests they wouldn't feel out of place among a crowd of such people.
In contrast, the people who prefer the term "ethics" in their work are grounded in real and present issues. They concern themselves with reasonable regulation. They worry about the current-day environmental impacts of training large models. In short, they are concerned with actual issues, rather than the alleged potential for a statistical model to "develop" sentience or exhibit properties of "emergent" intelligence (subjects from the annals of the science-fiction writing of last century).
I hope the author can clarify their choice of phrasing in their work, though I worry they have chosen their words carefully already. Readers should exercise caution in taking the claims of a soothsayer without a sufficient quantity of salt.
If this claim of yours was sustainable, your grievance with the writing style would make a lot of sense. But GPT-4 can clearly reason about novel problems which were not in its training data. I think you should consider reading some of the many citations in the document which describe examples of that happening.
Really, if it always produces output that conforms to "expectations of a reasoning actor", isn't that defacto a "reasoning actor"? What's the use of making the distinction?
I apologize for being difficult.
Who decided that this was the only thing that matters?
The thing that concerns me isn't that it answers questions in a way that resembles a human. What concerns me is that some people think it thinks like a human, which it does not, and writing about it in an anthropomorphic way exacerbates this problem.
The issue at hand is not the observable output of the system, but the way in which the system works — which we have knowledge of. It's not magic, it's not emergent, it's not sentient, and it's certainly not intelligent. It's a (very fancy and impressive!) predictive text system. Imbuing it with extra properties is disingenuous.
you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems.
From your citation itself:
> More broadly, it would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state?
I agree whether LLMs trained on natural language do develop world model is an interesting and open empirical question. They clearly can, but we don't know whether they do. I do consider the evidence pretty suggestive that they do. But this is clearly just an empirical question, not a question that can be answered by arguments as useless as "they are statistical models", because statistical models can and do learn world model.
I actually consider your considering this result not incredibly surprising pretty surprising.
LLMs are trained on the "rules" of human written language, which are sufficiently distinct from anything in the actual world that find world modelling would, indeed, be a surprise to me.
From TFA's conclusions section:
"Open debates over whether we describe LLMs as understanding language, and whether to describe their actions using agency-related words like know or try, are largely separate from the questions that I discuss here (Bender & Koller, 2020; Michael, 2020; Potts, 2020). We can evaluate whether systems are effective or ineffective, reliable or unreliable, interpretable or uninterpretable, and improving quickly or slowly, regardless of whether they are underlyingly humanlike in the sense that these words evoke."
As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Blake Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics.
To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so.
I found this recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0>
>Anthropomorphizing them is not just technically incorrect, but morally disingenuous.
Wat
In academia (where I am, though my specialty is a step removed), "AI" is exactly the region of research concerned with statistical problem-solving. Machine learning is sometimes synonymous, though I haven't been able to determine whether this is consistent among all self-described AI researchers or just some of them.
Systems like GPTs are not capable of "thought", full stop. There is no ambiguity. If this is hard for you to accept, I'm sorry, but it is a fact. They have no agency.
I saw someone sketch a good explanation. They were responding to a claim that Chat GPT could "pass the Harvard admissions process". While it is true that the system can access sufficient data to respond accurately to questions on an exam, and it could be prompted to generate an essay, it cannot of its own volition choose to submit an application. It doesn't even "know" that Harvard is a real place to which it can apply, nor does it "understand" how it could go about "learning" how to find out about these things it does not know. It simply isn't capable of these things. (And even if it were that wouldn't necessarily be evidence of intelligence, but for now it's a sufficient distinguisher.)
> Wat
Can you explain what about my last point was "wat"-worthy? Do you disagree that it is morally problematic to pretend a thing is sentient when it is in fact not?
We may just live in a world where there’s room for more than one viewpoint and people are able to research different aspects of the same thing without getting into debates over dogma.
I mean, inclusion -> https://news.ycombinator.com/item?id=34698769
AI safety is not a "viewpoint", it is a marketing strategy. The goal of the people leading that charge is to drum up FUD to increase investment in their useless snake oil companies. I feel no remorse for not being inclusive of their "perspective".
Thank you for a pleasant break!
https://www.patheos.com/blogs/wakeupcall/2020/05/10-thought-...
Maybe when the technology is physically capable of thinking for itself, rather than just being statistically likely to respond in a way that appears like the way a human would respond. Which may never happen, for what it's worth.
> I think LLMs are at the point where adopting an "intentional stance" (using Daniel Dennett's term) is reasonable.
I hadn't heard of this term so I looked into it, and wow, what a biased Wikipedia article it has. It's written with a completely partial stance. I've no idea whether to take it seriously, to be honest.
That said, I disagree strongly. LLMs do not "intend" anything, and pretending they do is (in my view) highly problematic. They're just (sophisticated!) prediction engines; nothing more.
What definition of reasoning do you have in mind such that LLMs don't do it?
While not learning, they can still run inferences on what they are fed as context within the token space. And because they are becoming so good at getting context out of structured data, they can use the context provided as a layer on top of their knowledge.
So you can at a very good approximation upload knowledge into them, and such knowledge can represent how to use tools to fetch data they don't know about - i.e. what langchain agents are doing.
Now of course that is not 'llm' some, is llm + code for informaron discovery, but that code is entirely supported and enabled by the existence of good enough llm
Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
Google gives the following definition of the verb "reason":
> think, understand, and form judgments by a process of logic.
LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more.
> LLMs obviously reason and understand by any evaluation that can be carried out.
Uh-huh. Sure, Jan.
"LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up.
They don't form judgement or conclusions? Sure looks like they do. So what's the difference ?
What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and "real" understanding or reasoning.
Such a huge difference I should be able to test for it. Don't understand how you can tell me what I'm seeing isn't real reasoning but fail to provide a way to empirically determine the difference.
>Uh-huh. Sure, Jan.
Yeah. https://arxiv.org/abs/2212.09196 Many other evaluations to carry out.
First the vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.
For the text, LLMs don't really have any problem with analogical reasoning https://arxiv.org/abs/2212.09196
The vision problems will require something much more than an image to text objective task. It will require the equivalent of an artificial visual cortex. We don't have that yet.
For abstract analogical reasoning, LLMs don't have a problem with that. https://arxiv.org/abs/2212.09196
If you're saying that we can't use a problem if any analog of that problem has ever been described, you seem to be arguing more strongly that it is a general intelligence than I am.
That’s not intelligence that’s computers having better memory than humans. Useful, certainly, but hardly skynet.
If you're saying that it is not possible to change the details enough to avoid the model being able to answer that type of question, I think you are admitting that the model has learned a generalized ability to answer questions of that class, and is not actually using its memory to answer at all.
I don't care about whether it learned that generalized ability from seeing examples of the question and answer, which it then deduced an algorithm for and generalized -- that's how most people learn most things.
As an aside, I’m really starting to hate these threads on here, people are constantly reading words that aren’t there in search of gotcha-it’s-skynet. It’s not. It’s just pattern matching and randomness with a giant amount of information encoded.
I'm not sure what to say, other than that if you'd like to have less frustrating conversations, you could do better than showing up with hearsay where someone asked a question they thought was unique, but it wasn't, and it can't be modified to be unique and then asked again, and you aren't willing to tell us what it was, and possibly don't know yourself.
It is not possible to have a serious conversation about your claim, and that's not because it is being intentionally misunderstood.
> skynet
You're the only person mentioning skynet. The conversation is about a ridiculous claim made up-thread that GPT-4 cannot reason or understand anything, which is disprovable within a few minutes of using it thoughtfully.
Complex behavior arises from simple systems all the time. You can't prove that these systems don't reason, no matter how loudly thou doth protest.
Instead, could you operationally define "reason" in a way that a human is, say, 90 % likely to pass the test and GPT is 10 % likely to do?
Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the equivalent of an artificial visual cortex. We don't have that yet
The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here.
For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196
We are talking about a model of language and our expectations for what the words would look like if a person were using language to reason with. People are confusing one thing (their own interpretation of what they are reading) with another (a statistical model that is being driven to align with certain specific human expectations).
The alternative is having 1000 different results for different kinds of arrows, and averaging out the results for the ones similar to the input arrow.
An LLM is working on text tokens, it’s trying to give the most statistically common next token based on everything that’s been fed into it. Does that statistical model abstract the objects and concepts it talks about? Eh? I don’t know
vvv
But, coming at this as *respectfully and curiously* as I can here:
^^^
I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source.
Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem.
So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
Your intuition about the architecture might be causing the distrust. Yes, the ends of the model are words in and words out (although GPT-4 is now multi-modal and can accept images as input too), but that doesn't mean the calculations in the middle are dealing in words. The "language" in "large language model" is misleading. It's a large generalized model that imports and exports language.
It all still feels rather “attribute magic to complexity” to say that there’s some human element skimmed off the patterns of our language, hidden in the coefficients somewhere.
I’m also not a magician, so really who am I to say :)
However, that feels a bit magical to me at the moment. Still interesting to be along for the ride humanity is going on!
Its actually a crazy idea that the only thing we needed to do to make LLMs more effective was just scale up the parameters
Though the transformer architecture definitely made it possible to handle the complexity
Is that true? Because LLMs are trained on the work of human: a LLM that learned all of the content of the articles would at best yield output equal to that of the human writing.
The reasons the author gives seem unfounded to me:
> First, they are trained on far more data than any human sees
If the human writer has access to google (everyone in modern society) this point is moot.
> In addition, they are often given additional training using reinforcement learning before being deployed.
If this is human in the loop RL, then the upper bound would still be the human training it. If it isn't, refer to #1.
There's also a difference between having access to and being able to 'comprehend' all previous works in a given genre for generating music, that a human simply won't have the ability to process and recall when creating their own music and the AI could surpass. Humans simply having access to information is different than using information.
But a single human can only read so much information in a finite amount of time (their life). Sure, the same thing applies to LLMs, but the theoretical limits are orders of magnitude higher.
And while this probably isn't what you were talking about, it's also pretty clear that any time-limited task favours LLMs. There's no human on earth that could write a non-trivial short story in 10 seconds, regardless of quality level.
Does not follow at all. There’s a vast difference between being able to search for and internalize a tiny, tiny sample of all the available information, and literally having sampled and synthetized all of it.
> If it isn't, refer to #1.
Meaning what? LLMs use adversarial learning, the same thing that allowed AlphaGo to reach superhuman levels in Go.
Would that enabled these models to compress themselves?