Simply explained: How does GPT work?

For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with.

(edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)

With this in mind, I think asking whether ChatGPT *in and of itself* is "conscious" or has "agency" is sort of like asking if the speech center of a particular human's brain is "conscious" or has "agency": it's not really a question that makes sense, because the speech center of a brain is just one part of a densely interconnected system that we only interpret as a "mind" when considered in its totality.

If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

I imagine many definitions are initially rather broad and only get refined down over time. Laertius gives us a classic example:

> Plato defined man thus: “Man is a two-footed, featherless animal,” and was much praised for the definition; so Diogenes plucked a cock and brought it into his school, and said, “This is Plato’s man.” On which account this addition was made to the definition, “With broad flat nails.”

I don’t think it’s correct to think of that as infinitely moving goalposts, however. More that the weakness of definitions isn’t always immediately transparent.

It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type.

For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it.

We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform.

Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT.

It must understand/have structure, or at least use a crutch to get it over the finish line..

Let's say we're trying to build a calculator that only needs to do integer addition. And we decide to build it by building a giant if-else chain that hardcodes the answer to each and every possible addition. And due to finite resources, we're going to hardcode all the additions of integers up to absolute value N, but we will increase N over time.

Everything you said applies equally to this situation: it quacks like a duck, and when we talk about things it can't do we have to continually move the goalposts each time a new version comes out. It also has the property that there is a "scaling law" that says that each time you double N you get predictably better performance from the system, and you can do this without bound and continually approach a limit where it can answer any question indistinguishably from something we might call "true understanding".

But I think it's a bit easier to agree that in this case that it's not "really doing" addition and is a bit short of our wish to have an artificial addition system. And if someone touts this system as the way to automate addition we might feel a bit irritated.

Again, many people will say that this is a bad analogy because LLMs operate quite differently, and I'm not trying to argue for or against that. Just trying to give my explanation for how a certain understanding of the facts can imply the kind of conclusion that you are trying to understand.

The pattern is counting the number of closed spaces in each letter of the spelled-out number. A closed space is any enclosed space in a letter, such as in the letters "a", "b", "d", "e", etc.

Following the pattern:

- one -> 2 (there are closed spaces in the letters "n" and "e") - two -> 1 (there is a closed space in the letter "o") - three -> 2 (there are closed spaces in the letters "h" and "e") - four -> 1 (there is a closed space in the letter "o") - five -> 1 (there is a closed space in the letter "e") - six -> 0 (there are no closed spaces in the letters) - seven -> 2 (there are closed spaces in the letters "e" and "n") - eight -> 1 (there is a closed space in the letter "g") - nine -> 1 (there is a closed space in the letter "e") - ten -> 1 (there is a closed space in the letter "b") - eleven -> 3 (there are closed spaces in the letters "e", "l", and "v") - twelve -> 2 (there are closed spaces in the letters "b" and "d") - thirteen -> 2 (there are closed spaces in the letters "b" and "d")

Each item follows the pattern, as the number of closed spaces in their letters matches the corresponding number in the pattern.

The whole sequence is:

one -> 2 two -> 1 three -> 2 four -> 1 five -> 1 six -> 0 seven -> 2 eight -> 1 nine -> 1 ten -> 1 eleven -> 3 twelve -> 2 thirteen -> 2 ...»

It is clear the model doesn't know what it is talking about.

Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding?

Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sadness, depression, love, etc.), a sense of what philosophers like Elizabeth Anscombe call I-ness, etc. Most of this stuff, to me, seems like is language-agnostic. Even a baby that can't speak feels pain or happiness. Even a dog feels anxiety or affection.

LLMs are a cute parlor trick, but a phantasm nonetheless.

This isn't true. If a plane flies like a bird and you only need it for flying it doesn't then follow that a plane is a bird "for all intents and purposes".

I am not sure they can, but the difference is profound and material. A machine that actually understands, like a human being, is not going to be (can not be) entirely truthful or transparent. There will be private inner thoughts, idea formation, and possibly even willful intent, as a direct consequence of understanding. And the nature of interactions, regardless of superficial similarity, shifts from one of utility to relationship. For example, we would care to know if e.g. the systems entrusted with apocalyptic deterent forces are mechanisms or organisms.

Please note that not a single one of us has ever interacted with any intelligent life form lacking a sense of self, or an ego. Thus, all our sensory registers of another 'intelligent being' are learned in a context of the implicit 'this other is like me'. We are not equipped to distinguish or articulate intelligence (in the abstract) merely based on sensory information. Note that even non-verbal communication, such as jabbing a friend in the ribs, are all learned to have a certain meaning in that very same context of implicits, and any mechanism that mimicks them (via training) will be afforded the same projection of the implicit. I do not believe there is, in fact, any non-destructive test of determining 'consciousness' in an entity. (Destructive, since there may be long running tests of a subject than can be shown to be probably accurate, possibly via creating situational problems involving survival, and unexpected circumstances.)

Ask yourself what is it that convinces you that the last person you spoke with (in real life) was actually conscious? I assert that the entire matter is a 'fictional certainty' based on assumption of shared nature. "They are conscious because I am".

Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html

I don't think an intelligence needs to be human, and it should be physically possible to create an intelligence which is synthetic. But in order to call the intelligence "general", and to rely on it for the purposes that designation implies, it would need to be able to successfully navigate the world, which requires access to that world and the use of the world as its own model, rather than the much simpler and coarser intermediary of text. In order to claim that an LLM can fully navigate the world after being trained on pure text, we would have to believe that all our writings across history have exhausted what there is to say about the world. This is not to say an LLM cannot be useful for some purposes, but there will be key ways in which they fail because they have no sense of meaning or what the world is like. Whether consciousness is required to solve this I don't know, but we simply haven't begun to approach a system that can meaningfully address the world as a world.

In the long run tech does a bit too well with "food in their belly" to the point that obesity is the main problem in the English speaking world.

As to programming it's quite cool getting chat GTP to write code and stuff. If you can't beat it make use of it I guess.

Seriously tho, taking some time to get away from it would be good. Ignorance is bliss, this too shall pass etc.

(btw nice piece of writing, you should do it more often!)

Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive, we may say, this is pretty much where our own demons live.

(I actually do think that revisiting Lacan in this context may be productive.)

It's a bit more plausible when we phrase it that way...

Who got caught for feeding some Rats ; Tremendous Work!

All the people tell me, many men, biggly men - many with tears in their eyes...

That I have done nothing legally-wise

But the truth is ; I am an enormous dork.

>>_Created by an actual Human Being with actual DNA for crime scene evidence._

-

But just when they tried to brush under a rug

To try to make the folks 'shrug'

Is the Streisand Effect as a scar

As everyone knows of payments to a Porn Star

And the nation will know youre a simple thug.

It's a bit like the Sagan quote: "If you wish to make an apple pie from scratch, you must first invent the universe".

Sometimes for GPT to "just" complete the next word in a way that humans find plausible, it must, along the way, develop a model of the world, theory of mind, abstract reasoning, etc. Because the models are opaque, we can't yet point to a certain batch of CPU cycles and say "there! it just engaged in abstract reasoning". But we can see from the output that to some extent it's happening, somehow.

We also see effects like this when looking at collective intelligence of bees and ants. While each individual insect is only performing simple actions with extremely limited cognitive processing, it can add up to highly complex and intelligent/adaptive mechanics at the level of the swarm. There are many phenomena like this in nature.

Aren't you just doing that?

I guess ... this is what confuses me. GPT -- at least, the core functionality of GPT-based products as presented to the end user -- can't just be a language model, can it? There must be vanishingly view examples from its training text that start as "Write a Limerick", followed immediately by some limerick -- most such poems do not appear in that context at all! If it were just "generating some text that's likely to come after that in the training set", you'd probably see some continuations that look more like advice for writing Limericks.

And the training text definitely doesn't have stuff like, "As a language model, I can't provide opinions on religion" that coincides precisely with the things OpenAI doesn't want its current product version to output.

Now, you might say, "okay okay sure, they reach in and tweak it to have special logic for cases like that, but it's mostly Just A Language Model". But I don't quite buy that either -- there must be something outside the language model that is doing significant work in e.g. connecting commands with "text that is following those commands", and that seems like non-trivial work in itself, not reasonably classified as a language model.[2]

If my point isn't clear, here is the analogous point in a different context: often someone will build an AND gate out of pneumatic tubes and say, "look, I made a pneumatic computer, isn't that so trippy? This is what a computer is doing, just with electronics instead! Golly gee, it's so impressive what compressed air is [what LLMs are] capable of!"

Well, no. That thing might count as an ALU[1] (a very limited one), but if you want to get the core, impressive functionality of the things-we-call-computers, you have to include a bunch of other, nontrivial, orthogonal functionality, like a) the ability read and execute a lot of such instructions, and b) to read/write from some persistent state (memory), and c) have that state reliably interact with external systems. Logic gates (d) are just one piece of that!

It seems GPT-based software is likewise solving other major problems, with LLMs just one piece, just like logic gates are just one piece of what a computer is doing.

Now, if we lived in a world where a), b), and c) were well-solved problems to point of triviality, but d) were a frustratingly difficult problem that people tried and failed at for years, then I would feel comfortable saying, "wow, look at the power of logic gates!" because their solution was the one thing holding up functional computers. But I don't think we're in that world with respect to LLMs and "the other core functionality they're implementing".

[1] https://en.wikipedia.org/wiki/Arithmetic_logic_unit?useskin=...

[2] For example, the chaining together of calls to external services for specific types of information.

With respect to interrogating GPT if it does something wrong - the reason why people do it is because it works. With GPT-4 especially, you can often ask it to analyze its own response for correctness, and it will find the errors without you explicitly pointing them out. You can even ask it to write a new prompt for itself that would minimize the probability of such errors in the future.

Regarding the speculation/destroy society, I was directly answering questions that I got from laypeople around me. The consequences on society I don't think are much speculation: it's going to have a big effect on many jobs, just like AI has started to have but much more. For the philosophical questions, I tried to present both sides of the issue to show that it's not just a clear "yes or no": some people will happily argue with you about GPT being smart/skilled/comparable to a human brain. Anyway, it's just an introduction to the questions that you might have about it.

It really depends on who the target audience is. There's been a lot of scare mongering in the news about it lately and I think the last part tries to address that. It first offers an explanation that my parents can understand and then addresses what they have been hearing about in the news.

So, I would say it is great to share it with them and I think they are the intended audience.

This seems to me to be obviously incorrect, and should be apparent after a few minutes of playing with GPT4. What makes it so powerful is how general-purpose it is, and it can be used for literally an unlimited set of tasks that involve human language. To say that it's not "smart" begs the question of what exactly constitutes smart and when you'll know that an AI has achieved it.

But it wasn't designed. It's not a computer program, where one can make confident predictions about its limitations based on the source code.

It's a very large black box. It was trained on guessing the next word. Does that fact alone prove that it cannot have evolved certain internal structures during the training?

Do you claim that an artificial neural network with trillions of neurons can never be intelligent, no matter the structure?

Or is the claim that this particular neural network with trillions of neurons is not intelligent? If so, what is the reasoning?

> It is not smart

"Not smart" = "not able to reason intelligently".

Is that a falsifiable claim?

What would the empirical test look like that would show us if the claim is correct or not?

Look, I realize that "GPT-4 is intelligent" is an extraordinary claim that requires extraordinary evidence.

But I think we're starting to see such extraordinary evidence, illustrated by the examples below.

https://openai.com/research/gpt-4 (For instance, the "Visual inputs" section)

Microsoft AI research: Many convincing examples, summarized with:

"The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence.

This is demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction), its range of topics on which it has gained expertise (such as literature, medicine, and coding), and the variety of tasks it is able to perform (e.g., playing games, using tools, explaining itself, ...)."

https://arxiv.org/abs/2303.12712

Programming languages are a whole lot more structured and predictable than human language.

In JavaScript the only token that ever comes after "if " is "(" for example.

It would convince a lot of people with the breadth, despite not really having much depth.

The real GPT model is much deeper than that, of course, but my toy example should at least give a vibe for why even a simple thing might still feel extraordinary.

- What is thinking, exactly?

- Does human (or superhuman) thinking require consciousness?

- What even is consciousness? Why is it that when you take a bunch of molecular physical laws and scale them up into a human brain, a signal pattern emerges that feels things like emotions, continuity between moments, desires, contemplation of itself and the surrounding universe, and so on?

- Why and how does a string predictor on steroids turn out to do things that seem so close to a practical definition of thinking? What are the best evidence-based arguments supporting and opposing the statement "GPT4 thinks"? How do people without OpenAI's level of model access try to answer this question?

(And yes, it's occurred to me that I could try asking GPT4 to help me make these questions more complete)

Anything it generates means nothing to the algorithm. When you read it and interpret what was generated you're experiencing something like the Barnum-Forer effect. It's sort of like reading a horoscope and believing it predicted your future.

It’s not like we all agree on what thinking is. We never have. It may not even be one thing.

When you initiate the model with some input where you expect some particular correct output, that means there exists some completed sequence of tokens that is correct—if that weren’t true then you either wouldn’t ask or else you wouldn’t blame the model for being wrong. Now imagine a machine that takes in your input and in one step produces the entire output of that correct answer. In all nontrivial cases there are many more _incorrect_ possible outputs than correct ones, so this appears to be a difficult task. But would you say such a machine is “thinking”? Would you still consider it thinking if we could describe the process mathematically as drawing a sample from the output space; that it draws the correct sample implies it has an accurate probability model of the output space conditioned on your input. Does this require “thought”?

GPT is just like this machine except that instead of one-step, the inference process is autoregressive so each token comes out one at a time instead of all at once. (Note that BERT-style transformers _do_ spit out the whole answer at once.)

It’s possible that this is all that humans do. Perhaps we are mistaken about “thinking” altogether—perhaps the machine thinks (like a human), or perhaps humans do not think (like the machine). In either case I do feel confident that human and machine are not applying the same mechanism; jury is still out whether we’re applying the same process.

There are two ways of looking at this.

1) In order to predict next word probabilities correctly, you need to learn something about the input, and the better you want to get, the more you need to learn. For example, if you just learned part-of-speech categories for words (noun vs verb vs adverb, etc), and what usually follows what, then you would be doing better than chance.. If you want to do better than that they you need to learn the grammar of the underlying language(s).. If you want to do better than that then you start to need to learn the meaning of what is being discussed, etc, etc.

If you want to correctly predict what comes next after "with a board position of ..., Magnus Carlson might play", then you better have learned a whole lot about the meaning of the input!

The "predict next word" training objective and feedback provided doesn't itself limit what can be learned - that's up to the power of the model that is being trained, and evidentially large multi-layer transformers are exceptionally capable. Calling these huge transformers "LLMs" (large language models) is deceptive since beyond a certain scale they are certainly learning a whole lot more than language/grammar.

2) In the words of one of the OpenAI developers (Sutskever), what these models have really learnt is some type of "world model" modelling the underlying generative processes that produced the training data. So, they are not just using surface level statistics to "predict next word", but rather are using the (often very lengthy/detailed) input prompt to "get into the head" of what generated that, and are predicting on that basis.

Of course maybe I'm wrong and it's AGI and it will find this comment and torture me for for insulting it's intelligence.

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

[1]: https://www.amazon.com/Deep-Learning-Python-Francois-Chollet...

[2]: https://en.wikipedia.org/wiki/Transformer_(machine_learning_...

To understand LLM from ground up, the following topics would help.

  - Machine Learning basics. e.g. weight parameters being trained.  
  - Neural Net basics.  
  - Nature Language Processing basics.
  - Word vectorization, word embedding. e.g. Word2Vec.
  - Recurrent Neural Net basics.
  - LSTM model.
  - Attention and Transformer model.
  - Generative model like GAN.
  - Generative Pre-trained Transformer.

The thing to remember is that GPT has no state apart from the context, so it can't "remember" anything apart from what's in the text. That doesn't mean it shouldn't be able to stay consistent in a guessing game but it does mean it can't keep secrets.

Some of that can be solved with layers above GPT where say it it told it can save "state" that isn't passed on to the human but fed back in to generate the next response. But the size of that context is very limited. (a few thousand words)

There seem to be a fair number of experiments playing with giving GPT this kind of long term memory, having it establish goals then calling it over and over as it accomplishes subgoals to try to work around those limitations.

In other words, it's seen (via its extremely large training set) that when asked that specific question, the response is most often a character from a particular set of characters, which happens to represent the numbers 0 through 100. It doesn't "understand" what that means in any real way. If the internet was full of examples of people answering "monkey" to that question, that is what it would sometimes answer.

Me: I want to play a game with you, you have a number from 1-10 in your mind. I try to get the number. If I’m right you tell me that I’m right, I’m too low you tell me so, if I’m too high you tell me too high. Let’s play!

GPT: All right, I have a number between 1 and 10 in my mind. Go ahead and guess!

Me: 2

GPT: Too low! Try again.

Me: 9

GPT: Too high! Try again.

Me: 5

GPT: You got it! The number I had in mind was 5. Well done!

The way it works is that each time it's tasked to produce a new response, it can view the entire history of the game. It knows that if it's said "higher" to 65 then it would be inconsistent to say "lower" to 64. Eventually this process terminates and the AI admits I "got" the number. The chat transcript up to that point is consistent with a "win".

What's wild though is that I can ask it to "regenerate" it's response. Over and over. Using this, I can convert a situation where a transcript which leads to a "too high" response into one that reads "too low". I'm, in essence, simulating fresh games each time and sampling over the choices of random numbers that GPT offers.

But it should also break the illusion of GPT specifically "having a mind". As I was chatting with it interactively, it was not really selecting a number but instead evaluating the probability of my particular guess sequence having the set of responses it actually saw. It then samples possible continuations. The more questions I've asked (and the more informative they were) the less variation remains in that selection of possible consistent continuations.

Or perhaps more consistent is the idea that within any single "call" to GPT to generate one further token (not even one further response) it may "have a mind", a particular choice of number, or it may not. It's actual behavior is indistinguishable either way. A whole chat dialogue, indeed even the rolling out of tokens from a single response it gives, are certainly (autoregressive) probabilistic samples over this process in either case.

(Edit, also worth noting that some evidence suggests GPT, including 4, is pretty bad at randomly drawing numbers.)

Do you see what I mean?

The bing version might run a bing query, fetch the X top pages, run GPT on it, return a response based on what it read, and in the back assign the summary to the source

Even then. I've had it write programs that were syntactically correct and produced plausible, but incorrect behavior. I'm really careful about what I'll use GPT-generated code for. IMO write the tests yourself, at least.

> Which is more similar, Apples & Oranges or Lisp & Almonds?

and it said:

> When comparing these two pairs, the more similar pair would be Apples & Oranges. Both apples and oranges are fruits, and they share many characteristics, such as being edible, containing vitamins and nutrients, and growing on trees.

> On the other hand, Lisp & Almonds are quite dissimilar. Lisp is a programming language used for computing and software development, while almonds are a type of nut commonly consumed as a food item. They belong to completely different categories and share almost no common characteristics.

And the question about which is better had a reasonable answer, too. This is GPT-3.5 but GPT-4 was roughly equivalent:

> It's not really possible to compare the relative merits of Lisp and almonds, as they are fundamentally different things.

> Lisp is a programming language, and like any programming language, it has its own strengths and weaknesses, depending on what you're trying to accomplish. Some programmers might prefer Lisp for certain tasks, while others might prefer a different language.

> Almonds, on the other hand, are a type of nut that are known for their nutritional value and health benefits. They are a good source of healthy fats, protein, and fiber, and can be a valuable part of a healthy diet.

> So, whether Lisp or almonds are "better" really depends on what you're looking for. If you're a programmer, Lisp might be better for certain tasks, while if you're looking for a nutritious snack, almonds might be a better choice.

Neuroscience is nowhere near finding out the connectome of a whole human brain so why not, we should look into these models as hints about what our circuits do.

I think what puts people off about these models is that they are clockwork: they won't even spit out anything unless you put some words in the input. But i can imagine adding a second network that includes an internal clock that continuously generates input by observing the model itself, that would be kind of like having an internal introspective monologue. Then it could be more believable that the model "thinks"

Maybe this ChatGPT stuff is "smarter" than I've been giving it credit.

Nowadays, IBM's Watson is simply a brand name for any AI/ML related products under IBM.