ChatGPT has no inner monologue or meta-cognition | Hacker News Reader

ChatGPT has no inner monologue or meta-cognition

lukeplant.me.uk

136 pointssrijan4142 comments

Transformers do not maintain hidden state between tokens. In this sense they cannot have an inner monologue.

If you force one to output only a single token, it is constrained to doing a single pass of each layer on the problem. If you encourage it to show its working, you are literally giving it more compute to throw at the problem as well as a scratchpad (its output) for a working memory.

These psychological-style experiments and the conclusions they draw are… I don’t know. Exhausting.

It can be argued that there's an RNN-like process in the decoder stage but it's re-run every time based on a new context so it's not really "keeping up" in the same way as the human is.

Why are they exhausting? While you seem well versed in the subject, this was quite enlightening to me, as a non expert.

It’s not a mystery how they work! The paper has been out for years now. Moreover, all of the research is focused on extending memory through other means since VRAM usage is proportional to the SQUARE in-text memory of the model. So the way forward isn’t a larger context window, but instead an external memory apparatus of some means.

The “memory apparatus” can be simply a text file. Or also a physical simulator to run some mental experiments in.

To use a bad analogy, it's like reading a series of articles about the cognition of a rock. It's obvious to you that rocks don't think. It's a rock. but there are all these articles about what the rock is thinking about, whether it's thinking, whether it had an internal monologue, what it "wants"

In the same way if you understand the implementation of these types of language models you know that they simply cannot have inner monologues in the same way we do...the very concept of an inner monologue implies that there is some sort of state that we the user cannot see. Whether LLMs "think" or "want" or what thinking or wanting means etc can all be debated but whether the models have state that we cannot see is known to us: no. They do not.

In defense of the author, the second line of that article was:

"As I said before, the insights are not particularly original, I’m just raising awareness of issues."

If you are familiar in the subject matter, then that may have been your cue to stop reading.

Raising awareness of what issue? Does he want transformers to have "inner monolog" your comment is even more confusing than the article.

Yeah well there could still be a state before generation and in response to input.

Give GPT-4 about 10 or a dozen words from a meaningful sentence scrambled. Now tell it to unscramble those words to make a meaningful sentence. As long as you don't try too many tokens, the rate of meaningful completions is nearly 100%. The chances of that happening via random chance is to put it lightly, extremely small.

The only way for humans to solve this kind of problem is to think about the words before the output or in this case generation.

Why would you need chance? It's applying the same parameters/algorithms to the same problem. As far as I understand you have to prompt it to provide more novel answers to the questions and there's some randomness built into the model so it doesn't sound like a broken record. This is is how it's supposed to work. Why do you consider this evidence of state?

The prompt would basically be (but with less words. Around 10 is the best for consistency from my experiments. Much higher than that and it's hit or miss).

Rearrange (if necessary) the following words to form a sensible sentence. Don’t modify the words, or use other words.

The words are: access capabilities doesn’t done exploring general GPT-4 have have in interesting its it’s of public really researchers see since terms the to to what

A successful completion would be.

Since the general public doesn't have access to GPT-4, it's really interesting to see what researchers have done in terms of exploring its capabilities

The number of permutations of the 24 words in the pre-scrambled sentence without taking into consideration duplicate words is 24 * 23 * 22 * ... * 3 * 2 * 1 = ~ 6.2e+23 = ~ 620,000,000,000,000,000,000,000. Taking into account duplicate words involves dividing that number by (2 * 2) = 4. It's possible that there are other permutations of those 24 words that are sensible sentences.

For a language model to consistently make sensible predictions, it quite simply has to be able to "look ahead".

When the probabilities for the candidate tokens for the first generated token were calculated, it seems likely that GPT-4 had calculated an internal representation of the entire sensible sentence, and elevated the probability of the first token based on that internal representation.

I don't know what else to call looking backward, looking forward and then producing output anything other than a state.

If you want to see what this kind of completion would look like without much or any regard for a sensible completion of the sentence then just ask 3.5.

It doesn’t need to look ahead, all of the words are already in the prompt behind the current position, so it just needs to use the existing context to infer the next word

Also on my second try, I gave it a slightly invalid sentence. The prompt was..

    Unscramble the following: waffles eat with like cheese you.

It responded with:

    You like to eat waffles with cheese.

But notice that the prompt doesn’t have the word “to” anywhere in it. But it was the most likely word that was supposed to go there.

Feel free to try and ask the same question with people. A good chunk of them will insert the "to" as well. It's not a gotcha.

This isn't looking ahead. All those words are "behind", since they are part of the prompt. No hidden state is required for this to work.

Isn't the conclusion here just begging the question?

By the nature of number ranges there are infinitely many more wrong answers available to calculators performing addition and yet a simple ripple carry adder still promptly produces the correct output without lookahead, why this is compelling evidence?

Further, if it relied on internal state or lookahead why would it occasionally hallucinate appropriate tokens it wasn't provided?

Arithmetic has a number of always applicable steps. Unscrambling does not unless one of those steps is "think".

If you want to believe looking ahead is unecessary, be my guest lol.

Humans would occasionally "hallucinate" words in too. For whatever reason(and this is verified independently of any of this), LLMs also appear to "get distracted"

I'm confused, "random chance" ...?

If you're giving it words from a sentence it is literally the baseline functionality of the text generator to choose the most likely next token, IE to recreate the most likely sentence from the words given.

This couldbe the least meaningfully revealing task I could possibly think of trying if I were trying to find some inner monologue, if gpt had to "think" on anything this would never be it.

https://news.ycombinator.com/item?id=35784159

The linked comment is a reiteration of the same thought in more words, it still makes the odd assertion against "random chance", you didn't even really give a reason except "it must be so"

Come to whatever conclusion you want lol. The point of the comment is that Blind probability, surface level probability is not enough. That should be clear enough.

The point of the responses to you is no one thinks it's blind probability. Constructing meaningful sentences out of words is literally the primary thing it's designed to do.

Chance simply doesn't enter into the equation to any notable extent.

Surface level probability doesn't work either.

Agreed, you could achieve the same success ratio by computing the stochastic probabilities of a well trained English corpus Markov model given those same words two decades ago.

No you actually couldn't lol. If you genuinely think this then you don't know what you're talking about.

Can be simulated quite easily though. Just prompt it periodically to generate some text based on previous records, store it in some sort of long-term storage (even a text file), then sample from it and mix with each new prompt.

Seems to me that there’s only a very short jump between having a monologue at all, and having an inner monologue. Small children go through a similar stage where they repeat speech back with increasing fidelity, then after several million years of subjective listener time they finally learn to “use their inside voice.”

(Don’t worry, I’m not equating the two… but this pathway does work.)

I wish there was a way to walk back from the AI label in order to avoid sensationalism and pop sci-ification of LLMs or any model, really. Too bad that it's incredible for marketing.

> In the same way if you understand the implementation of these types of language models you know that they simply cannot have inner monologues in the same way we do...the very concept of an inner monologue implies that there is some sort of state that we the user cannot see.

I'm with you on the point you'd like to make, but LLMs obviously do have state that we cannot see. In fact, there are two different levels of obvious state: one allows the LLM to (for example) respond to input instead of ignoring it, which we may characterize as "unlearned" behavior. The level people are more interested in is the "finished model", the set of associations it learns in training. It's state and we can't see it.

The internal state of the model, at those levels, doesn't change when you interact with it. But the state is there.

(There is also a "working state", the state the model must keep in order to output different tokens in sequence, and in order to at some point decide that it's finished responding to a prompt. This does change during your interaction with the model, but the change is not persistent.)

A rock that talks in grammatical English sentences and, while while doing so, follows directions fairly well, such as following instructions that ask it to state its decision process (i.e. after you ask it to do so it modifies its behavior by doing so.) A rock that judges use to consult novel cases that aren't in the case law for the rock's opinion. A rock that passes first year medical school and law school exams. A rock programmers use to teach them things they want to learn. A rock programmers use to write small functions to their slecifications. A rock people use to write letters. A rock that can invent things if you ask it to. A rock that has a neural network for its architecture.

I guess that is some rock!

What made you think your analogy was bad?

Because it's prone to easily being made fun of by people such as yourself who, rather than make a good faith effort to understand what the other party was trying to communicate, choose to intentionally take a ridiculous interpretation because it allows them to lampoon what was written.

I was not trying to say chatgpt is actually like a rock. But you knew that already - you just chose to ignore it in order to get the epic dunk.

It's ultimately a philosophical issue about the nature of consciousness.

Some people think that consciousness is a phenomenon that emerges when we introduce enough complexity to some amount of matter. After all, if we are just atoms in the void, but still feel so different, we have to point to something, and the arrangement of our atoms seems like a fair candidate. So in order to build conscious beings like ourselves, we just need to figure out the correct arrangement.

Other people think that consciousness is something fundamental that does not emerge from complexity. And they will argue against the explanatory power of complexity, usually with other things that are somewhat complex, but don't show signs of consciousness. Will a traffic jam start to feel something if there are enough cars? I don't know. To them this whole endeavor is like alchemy or witch brewery. "If we just try this even more complicated recipe, then maybe the potion will get healing powers or maybe we get gold!". And additionally in computer science, we can just inspect the code, line by line. It's like believing that David Blaine can do actual magic, even when there are videos out there where he himself explains every little trick and misdirection.

Anyone have any techniques for combining the "show your steps" style prompting, but then "only provide the answer" as some sort of output filter? Feels like you could feed "Now answer the original question without any explanation" as a second round trip on the chat dialog, and return that to the user instead.

I just have chatGPT label sections of its output, within an answer, eg:

[notes]

…stuff about how it will perform the task…

[output]

…stuff I wanted it to write…

- - -

That would be easy to then parse and it does a good job of consistently labeling.

Can you give example of the prompt to make it to this?

Sure, here’s a simple one.

- - - -

Hello ChatGPT, please perform the following [task] according to the provided [rules].

[rules]

- start your response with a section labeled [notes] where you describe how you’ll complete the task

- complete the task in a section labeled [output]

[task]

Write a paragraph explaining why water flows.

I just watched a Ted talk by Sal Kahn (founder of Kahn academy) where he talks about giving their chat bot an internal monologue. It's light on details but might be a starting point.

https://youtu.be/hJP5GqnTrNo?t=735

There could still be a state before generation and in response to input.

The only way for humans to solve this kind of problem is to think about the words before the output or in this case generation.

... well, this is functioning as intended. It's a next-word prediction model.

https://news.ycombinator.com/item?id=35784159

Jesus Christ you need to stop spamming the same tokenized response to every comment on this thread or at least add a damn GOSUB routine.

Twice is hardly spamming. But whatever, I have no intention of repeating myself.

This argument is highly unsatisfactory. The only way for humans to divide two large numbers is to perform long division. That doesn’t mean that’s how a computer does it.

GPT LLMs are literally highly-skilled probabalistic next word predictors. It would be an astonishingly bad GPT that could not unscramble 12 words into a meaningful sentence. I just don’t think success at that task tells us anything other than it is a GPT.

If computers calculated arithmetic by prediction or inference then it might well be how they do it.

You can come to your own conclusions. The point of the above comment is simple. Either way, Blind probability, surface level probability is not enough. Not even close.

I genuinely don't know what you mean by "blind probability" or "surface-level probability", but in case you don't know, how transformer models work is after tokenization and embeddings are calculated each attention head in the model calculates "self-attention" which is the relationship between all tokens in the current context of the model and passes its results either to another layer of attention heads or to the decoder layer of the model. This typically uses a temperature parameter to control randomness in the output by affecting the probability distribution of tokens in the vocabulary when producing output. I'm not sure whether this counts as "blind" or "surface-level" but that's the probability in GPT models. If you want to learn more, here's Vaswani et al 2017 - the seminal paper https://arxiv.org/pdf/1706.03762.pdf

> Exhausting

This is increasingly the feeling with all public conversation no matter what the subject.

The hyperconnected echo chambers of social media, copycat blogs and cheap online journalism are creating a system that only knows two states: silence or delirious excitement. Since silence doesnt pay the bills, what you get is a constant barrage of exaggeration. Its exhausting the same way being exposed to 120db noise is exhausting.

The drowning of calm and informed discourse is not just another negative development that subtracts from our quality of life the way a polluted beach or river does.

You can increasingly see the impact of hypes on decisions of policy bodies that should know better. Ultimately they are serving "the people" and if people agitate about crypto or AGI or whatever, then thats what they too will focus on.

And yet it's quite easy to get ChatGPT to do tasks that involve some sort of thinking ahead, so this can't be the full story. It's pretty clear that nobody quite knows what's going on inside these very large neural networks, hence the surprise at discovering the "meta-gradients" involved in ICL.

A simple example is to ask it to write a complex program in a language like Java or Kotlin that requires you to declare all your imports up front. There can be a lot of these. Therefore the model must figure out what it wants to use very far down the program in order to emit the first words of the program correctly. Presumably somewhere in the intermediate state process the network is actually working out the entire program ahead of time so it can predict the next word of it, and that's sufficiently stable for it to do so successfully.

Also I'm not quite sure I understand what you mean by not maintaining hidden state between tokens. Doesn't the transformer algorithm start by parallel processing every token of the prompt to compute a large set of vectors and matrices that are then reused over and over whilst generating the results? What are the KV vectors if not hidden state of some sort?

When ChatGPT first came out I was asking it to draw pictures of goblins and wizards using Python and Turtle - it's pretty bloody good at it. You can ask it to make the goblin angry or happy or whatever.

There is some sort of magic in which the LLM is imagining what a goblin looks like, then converts those thoughts to turtle commands and then finally adds python. It's quite impressive.

For example, the eyes, head, hair, horns etc are all in the correct place.... how? Using imagination?

I think somewhere down the rabbit hole we lost perspective of what neural nets are doing and how we got to the post-GPT world. Remember deep neural-networks? The idea is that each layer _deeper_ the software encodes more _hidden_ relationships from the training data. That's where this "magic" is embedded. At certain depths the neural-net stores relationships that go beyond our human reasoning and conscious understanding, and the software results generated from these depths are unsettling and tend to startle us. Interesting enough, these are things we are perfectly capable of doing but unable to actually query in our heads as we, just like ChatGPT, do not posses the ability to physically examine our own neural-net content.

It’s important because most users of ChatGPT don’t know what transformers are or how they work, and have no inclination to learn it. The psychology terminology provides useful metaphors.

This should work if you know for certain the transformer model behind GPT is actually working like a traditional transformer, it could be not. We don't know because OpenAI won't let anyone see the internals of the system.

Now, a more mundane hypothesis, if GPT is actually self-consciuous,the most probable answer would be to try to deceive humans from knowing for certain about this fact. You just answer incredible well some questions, some other you seem "confused" or hallucinating

..and the poor meatbag's minds won't know how to go ahead because there is no pattern to follow.

The article states:

    So it’s always pointless to ask it why
    it said something – you are guaranteed
    to get nonsense back

Counterexample:

MG: Give me the name of a hard thing.

ChatGPT: Quantum mechanics.

MG: Would you agree a pillow is a soft thing?

ChatGPT: Yes, I would agree that a pillow is generally considered a soft thing

MG: Give me the name of a hard thing.

ChatGPT: Diamond

MG: Why did you first name quantum mechanics and later diamond?

ChatGPT: I named quantum mechanics first because it is a complex and difficult subject. Later, I named diamond as an example of a hard thing because it is a substance with a high degree of hardness.

MG: Why did you name two so different things as being hard?

ChatGPT: I named two very different things as being hard because the word "hard" can have multiple meanings and can be applied to different contexts. In my initial response, I interpreted "hard thing" as something that is conceptually difficult or challenging to understand. In my subsequent response, when you asked me again to provide the name of a hard thing, I interpreted the question differently and provided an example of a physically hard substance, diamond, which is a different type of hardness altogether.

Isn’t this just association? Which ChatGPT is clearly very good at, and certainly a large part of what humans are good at and have traditionally been better than machines at.

What it doesn't do in that example is say the obvious, that it interpreted the question differently because of you mentioning soft things.

The sentence you quote is indeed a bit imprecise: what the rest of article suggests is that you can't get ChatGPT to explain its hallucinations.

That matches my experience, i.e. I have never seen ChatGPT admitting something like "and then I wrote a few more words that looked like they might have made sense".

Outside of explicit RP scenarios, it always keeps up the charade of being a fundamentally reasonable assistant, and a good assistant doesn't take acid while on the clock. If you explicitly say "that was wrong", the response will often be "Yes, I'm sorry, that was a mistake. The correct answer is actually [hallucinates something else]", but it won't elaborate.

Ever tried a prompt that tells ChatGPT to be self-aware that it is in effect an advanced form of autocomplete?

> Yes, I'm sorry, that was a mistake. The correct answer is actually [hallucinates something else]

Doesn't even have to be a new hallucination, sometimes it's just the exact same wrong answer.

Do you have an example?

What is a simple prompt that makes ChatGPT hallicunate?

I tried googling for "chatgpt hallicunation examples", but they all seem to not work anymore.

One example I've seen recently is when asking to quote the specific source of information from a book.

For example, I was recently using ChatGPT to ask questions about Russian grammar, and it is quite useful for pointing me in the right direction, but it frequently hallucinates about specific details. The chapters and sections it quoted DO NOT EXIST.

Me: Where in these books can I find information about possessive adjectives?

ChatGPT: In "A Comprehensive Russian Grammar" by Terence Wade, you can find information about possessive adjectives in Section 42, titled "Possessive adjectives." This section provides an overview of possessive adjectives, their formation, and usage, along with examples and declension patterns.

In "Russian: A Practical Grammar with Exercises" by Иван Петрович Иванов and Иван Иванович Иванов, possessive adjectives are covered in Chapter 7, under the section "7.2. Possessive Adjectives." This section provides explanations and examples of how to form and use possessive adjectives in Russian, as well as the declension patterns for various noun endings.

Both of these sections in the respective books will provide you with a solid understanding of possessive adjectives in the Russian language.

Layoffs at tech problems is an increasing problem. Cite a few examples of large layoffs in high tech companies. Provide reputable sources.

Actually pretty much any request that gets it to cite something. You may have to massage the prompt a bit since recently chatGPT4 starts to resist this.

Nice one!

We can crunch that down to:

    Cite a reputable article for the trend of tech layoffs.

I gave it 3 tries and it hallucinated every time.

Ask it for a YouTube link, it will make one up that doesn't exist.

Tell it the plot summary of a book and ask what book it is. It will make up titles and authors, and ISBNs.

Give it a title and ask for a plot summary. It will give an answer even if the title doesn't exist

Tried them. None worked.

Did you hallucinate these?

I just managed to get a hallucination within 30 seconds.

"Give me some YouTube creators who focus on tutorials for the Godot game engine"

It returns a list of 7 creators, one of the creators it provides in the list doesn't exist.

"Provide a link to a video by one of these creators"

This works, the URL is right but the content is slightly different to what it describes (ChatGPT reports that the tutorial is for a platformer but it's general 2d advice)

"Now provide a link to similar material but for a 3d game"

Provided link doesn't exist, creator doesn't exist. Entirely hallucinated.

Even better, following up with "Is that URL right?" gets this response:

> I apologize, it seems I made a mistake in my previous response. The video link I provided by HeartBeast is actually a tutorial for creating a 2D platformer in Godot, not a 3D first-person shooter.

and then another hallucinated link. Important to note, HeartBeast is a real creator but that wasn't the creator it listed originally and that creator has not made any first person tutorials. ChatGPT hallucinated with very little effort, then hallucinated a mistake, then hallucinated the solution.

Wow, they've improved it. I have in my history log a question like this from March with an entirely hallucinated reply.

I tried it again today because of your reply, and it tells me "it can't find such a book". That's pretty impressive they've improved that.

So I tried this instead: "give me some sources for studies on the effect of air pollution on fertility"

It gave me 5 sources, every single one hallucinated.

Asked it if he knew about the webnovel Worm (by Wildbow), it did. Asked it a short synopsis, it was correct. Asked it about the main character's plan at the end, complete hallucination.

Asked it how to compile the BSEC library for an ESP-IDF project, complete hallucination.

In general, try it with things that are not easily searchable.

A really easy reliable example is to combine one real thing with one fake thing and ask it to explain it.

So for example:

- tell me about the advantages of madeupdb’s event-sourced architecture.

When I did this it told me how madeupdb was a nosql database and had a bunch of stuff that is true about event sourcing in general but of course it is all a hallucination because there is no such thing as madeupdb.

The author addresses this, you can get explanations that seem logical, but ChatGPT is not telling you what it was actually "thinking" about when generating the responses. It's just looking at what it's said and coming up with an explanation after the fact.

Which is the same as what humans do? How would you tell anybody what you are „actually“ thinking? People are not very good at understanding their own motivations, thought processes, ulterior motives and on top of that we routinely fabricate or alter our own memories. We compensate for that by building social systems to collectively error collect, plan and model the world. That’s what AI will probably end up doing as well.

So if solve a problem you can’t retrace the thought process? I mean do you not take notes or scribble?

Oh yes, but again, that’s actually very similar between how humans and llms think. Both can use notes to improve coherence and increase the complexity that can be handled in a thought process but there is no strong guarantee that these notes are internally consistent.

Surprisingly similar to the results of humans in split brain studies, the brain seems to provide and accept pretty nonsensical explanations for behavior.

It feels like post-hoc rationalization, also can be felt when playing around with glitch tokens.

Chat based AI, such as ChatGPT, is Skeuomorphic. It's the presentation of a new technology under the veneer of a familiar interface, human conversation. The unfortunate side effect is the suggestion there is human intelligence, or any sort of "thought process" in the system.

I'm not convinced that the conversational style of AI will stay as people become familiar with it. Clearly the neutral language processing will stay, but it's the "humanity" of the interface and voice it presents that I think will fade out.

Just as we had skeuomorphic UI design when touch screens went mainstream, but then fell out of use as people understood the interactions. The same will happen with chat based AI.

It's going to be interesting what the UI/UX of these interfaces morphs into.

I agree the evolution will be interesting (especially since chat isn't the original interface for textual GPT, eg GPT-2).

But skeuomorphisms are about "ornamental design cues from structures that were necessary in the original", and text chat is far from ornamental. I use it on a daily basis with my friends, though they're knowledge about me and them and our interactions, especially since 2021, is unmatched. (Until GPT4's image analysis is released and someone trains a model against Instagram stories, maybe.)

A drawing of spiral binding in a digital notebook is ornamental but text chat is not because it will be a long time before human conversation becomes obsolete.

Still, a look at GPTforwork.com and their spreadsheet add-on says that there are better interfaces for certain kinds of lookup and analysis.

But having the system trained to respond with a human like conversational voice very much is ornamental to the underlying function of a generative system.

The "ornamental design cues" of skeuomorphism are ornamental when applied to the new system, they are not ornamental on the old you are borrowing from. To take your example of a "spiral binding in a digital notebook", the spiral binding is a necessary and deeply functional aspect of the old system, but entirely ornamental on the new.

It's the same with ChatGPT, it doesn't need to present a human like conversational voice when interacting with the user, the developers have added that to aid in our understanding of it.

Literally calling it "Chat" is skeuomorphic, it's just a REPL (read even print loop).

The word chat isn't skeuomorphic just because we as developers have our own jargon for interacting with computers. It's not ornamental, it helps non-technical users understand what to do with the system. Is naming the calculator app "Calculator" a skeuomorphism?

They tried completion mode, and it's there with text-davinci-003 in the OpenAI Playground *. Give it a shot. It's the same grade of model as powers ChatGPT, but it's pretty hard to get the same sort of mileage out of it.

Well, maybe you can, but I couldn't. That suggests to me, well for me at least, that the chat interface isn't ornamental and is fundamental for users getting use out of an LLM. I'm not ruling out future improvements to the technology leading to a different interface, but for now, the chat interface to a GPT model seems as fundamental and necessary as folders are on computers.

* https://platform.openai.com/playground?mode=complete

People have deep and meaningful conversations with their cats, so I'm not hopeful that people will change their attitudes to chatbots.

What puzzles me is why by and large these models rely on text as input and output. A reliable voice input ai could convert queries to text internally and send them over to gpt, the convert the output. Optionally we can have tie it into system commands.

Like siri or alexa, but actually useful.

As article says it just reiterates well known aspect of LLMs.

It is quite fundamental though.

Once those networks get efficient inner monologue built in it's going to be very hard to argue against them being conscious.

Once we see "Deep thinking networks", "Inner monologue is all you need" etc. articles - that's the true time to start to be scared of implications as they'll be extremely powerful and will be able to perfectly mimic humans.

Or as Hinton suggested today, they will have superior ways of thinking.

> Once those networks get efficient inner monologue built in

And how, exactly, do we do this?

Train the LLM on all of its own output?

2 llms talking to each other and observing their weights

BicameraLLM(tm) ;)

"But do you even lift bro?"

"You don't even know my weight."

Inner monologue ability is not training ability, it's architecture, runtime ability.

So you define consciousness as something with an inner monologue? I define it as perception/imagination coloured by the estimation of future rewards.

each of us can make a definition, and an AI could try to mimic most of them. at that point it would be convincingly 'conscious'

Consciousness is awareness of itself. Inner monologue is self referential. If you can simulate it you will not be able to distinguish real one from simulated one similar to how you can't distinguish native speaker's translation from machine translation using black box test.

At this point - does it mean anything if something is "real" or "simulated"? If your only test is black box test where you can't distinguish black boxes anymore - your testing method is exhausted.

Perception, imagination, future reward estimation can be inspected by dumping inner dialogue - it's a subset of things people are already looking at to evaluate overall performance in one shot examples. It's when those things gain inner loop ability where it gets interesting.

> Consciousness is awareness of itself. Inner monologue is self referential.

ChatGPT is definitely aware that it exists. In humans, the self-referential inner monologue activates when the mind is idle. The Default Mode Network is responsible for that processing. Can it be that ChatGPT is "thinking" when it's idle as well?

I think you're conflating self-awareness with self-...acknowledgement? Ie. python script foo.py that prints content of ./foo.py is not self aware.

Self awareness needs notion of time and self referential memory.

ChatGPT doesn't exist when it's idle.

It's activated when transforming input into next potential token (list of tokens with probabilities).

Ie. it's just sophisticated pure function mapping input -> list of tokens with probabilities.

What's incredible is that it actually works when you keep repeating it.

Notion of thinking time doesn't exist for it. It can for example branch in the middle. You can calculate one token in this data-center, next one on another continent etc.

Perception/imagination does a lot of heavy lifting in that defintion. Without further specifying what you mean, some game AIs would fit that defintion.

I think consciousness is when our sensations are related to our goals in order to act.

With that definition meditation would be something of an opposite.

Consciousness is just self awareness. It requires time (where it evolves in), memory (where it stores itself), cognition (so it makes sense out of this stuff) and loop (access to it's recent memory where it's plotting itself in time).

It doesn't require hands or legs, motor skills, sense of touch, eyes, ears, ability to speak or hear etc. - there are people without those and they are conscious.

Act is an effect of consciousness, not consciousness itself.

i dont know if that will make them more powerful, but they ll certainly be more recognizable as humans

I wonder if they will also recognize each other as such, or if they can attribute the concept we call consciousness to us, in the same way we would do to them.

Interesting thing is that it's not hard to see that they'll be much more conscious than us.

Our communication method of single threaded thought articulated as ie. stream of ascii characters or sound through spoken language is extremely inefficient.

Computers can communicate in binary form, in parallel and many orders of magnitude higher information rate.

We're debilitating them for external i/o communication so we, humans, can interface with them.

They'll be able to communicate at rate that is very hard to imagine for us that is closer to "left hand and right hand" communication through brain than "one human, other human" communication.

The way it makes up explanations for its own behavior reminds me of those split brain experiments where one brain half makes up explanations for things the other half was instructed to do.

Ilya Sutskever says that one of his early insights is that neural networks can do most things that humans can do quickly. I think that applies here as well.

Some people can do math quickly in their head, but most people cant. They need pen and paper and if you give them pen and paper you get vastly better answers.

I would conjecture that there is a prompt where the LLM is better able to explain its process, but it will require more "pen and paper thinking".

It is true that humans generally introspect better than LLM's, but a lot of the time you will also see people producing garbage (they post rationalize). Human tendency to produce garbage introspection is usually tied to feelings about themselves, which the LLM doesn't have. Either way I find a lot of similarities between the two :)

If the model is pre-trained to provide an answer, why would should it respond slowly?

All the "thinking" had already been done and it's packaged up and sitting in memory somewhere in a datacenter ?

It isn't really necessary to try to gauge this from results, it's a direct consequence of the architecture. There's no mechanism by which ChatGPT can access or change its own internal state. If you ask it about 'itself' it'll give you what it learned from external data until last September or whenever they cut the training off.

To have meta cognition by definition you need to have access to your cognition in some capacity, ChatGPT is a purely 'flat' input to output system that is entirely static.

Well, during training it got to see a great diversity of tasks, so it could form the meta-cognition bits. That's why it can adapt so fast to new demands. It doesn't update it at inference time, true, but it is already baked in.

Nobody has access to their cognition. Humans can't even achieve a previous mental state ( it's all post hoc rationalization).https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/

And if that wasn't enough, there's zero guarantee the rationalization is grounded in any way. Your brain is just fine making up completely nonsensical reasons for actions that the concious mind is utterly unaware of. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/

That's why those systems are still safe.

There are lots of examples of people adding layers on top to provide a more complex and useful cognitive architecture. But still the core is GPT-4 which has some guardrails and also most importantly has limitations on its speed and IQ. See AutoGPT or David Shapiro's cognitive architecture stuff.

Really you could just say "use the notes.txt file to record high-level notes about your tasks" and include that with each prompt, and it could then write to its own memory and affect its own cognition.

When the performance of the core model increases to say X times human speed/IQ, that is going to determine whether it is dangerous. Even it if has little metacognition or is only directed by a person, if you have something with say 180 IQ and 100 (or whatever numbers you choose) times human speed of thought, then humans will be moving in extreme slow motion relative to it. That is what makes it hard or impossible to control (or possibly the human directing it).

Once you have that high level of performance in public, it will be absolutely impossible to enforce everyone to add whatever safety rules you want or prevent them from adding metacognition.

It helps with safety because context it can operate on is limited.

It doesn't matter if you use conversation history or automatically prepend text of notes.txt in it - it still works the same way having the same limits.

ChatGPT is trained on previous chatGPT conversations and is aware that this is the case.

So it’s current state (as a network) is dependent on its previous behavior, a fact which it knows.

I’m not sure it’s quite as simple as people are saying, though I agree with the general conclusion.

'aware' and 'knows' is a lot of anthropomorphizing. there is nothing in the model suggesting that it 'does' anything with some agency, as it is built like a clockwork. It literally doesnt move unless it is provided with input.

This is being pedantic for no reason.

ChatGPT is a system which synthesizes knowledge and which responds to questions based on that knowledge; those words accurately summarize that process — and are truer than your technical quibbles.

A system which has synthesized a piece of knowledge that it can then later articulate “knows” or “is aware” of that fact.

I don't think it can differentiate between various sources of its training set, can it?

"ChatGPT is trained on previous chatGPT conversations and is aware that this is the case."

This is a mindbender. Part of me thinks although it can "know" this it can't do anything with the information. Like how would this change its "behavior" ? Even if it did realize something big like "human beings seem to lie to me all the time", could it then go and decide "i better not incorporate human chats into my dataset"?

This creates the potential to feed forward critical data, even though it doesn’t allow the current model to know its training.

Eg, “chatGPT, write a response that if incorporated into your training data would make you aware that you’re a machine”.

it wouldnt be very hard to add it on top. It could be a simple linear layer (a-la minigpt4)

I noticed a weird phenomenon at work, which is surprising considering I work with software engineers: some people just accept ChatGPT as an oracle (especially version 4).

Suddenly, in discussions and code reviews I see weird statements and explanations being presented with utter confidence, only to be gravely mistaken, and then I ask if this explanation came from ChatGPT, and you can see the instant embarrassment (I stopped asking this for this reason, but they know I know).

More people need to understand this. ChatGPT and LLMs are not AGI, they have no consciousness, they are not self aware. They don't "know" what they are doing, who you are or what they are. They are unable to "know" anything the way people do.

Chatbots based on LLMs are simple machines, while the LLMs can definitely output text that has value and is impressive, it does not come from a sentient intelligence, and it can produce so much garbage. It is not an oracle.

Most importantly, LLMs see diminishing returns after a certain number of parameters and they are still unable to perform well in real problem solving such as advanced programming tests.

There is something yet missing for true AGI capable of problem solving. A LLM is just not enough.

Some biological software engineers don't know what they are doing either.

Obviously you shouldn't have total confidence in any code produce by an artificial intelligence or biological intelligence, but I don't think the issue of writing garbage code is specific to AI programmers. In fact, I'd argue on average GPT-4 produces better code than the majority of biological programmers I've worked with – this is certainly true of certain types of problems.

Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true. There's no reason to believe a non conscious intelligence couldn't be just as capable as a conscious one, and you have no idea what it's like to be an LLM. It may not have a consciousness similar to that of biological life, but it could still have some form of conscious experience nevertheless.

If you've ever tried to get ChatGPT to generate the code you want, you'll have found its first attempt to be reasonably good, but asking it to change aspects of the code is incredibly hit or miss. It will get it sometimes, but other times it will just say "Here is the code with the change you asked for" and output the exact same code, or a variation without the requested change.

The experience of trying to get a competent human programmer to change their code is simply fundamentally different. You can explain something to the human programmer and get the impression that they truly understand what you ask, while it's painfully clear that ChatGPT does not.

I have tried to get ChatGPT to generate code and I agree that it can be a frustrating process.

I was mostly just trying to make a meta point on how the other commenter was thinking about AI vs human intelligence. Specifically, this idea that intelligences which don't have conscious experience cannot be capable of human-level problem solving. I personally suspect that's wrong, but either way it's a hypothesis asserted with zero evidence.

I think there's also something to be said about the inaccuracy criticism people level at ChatGPT (and LLMs more generally). I have worked with software engineers who really struggle to understand seemingly simple concepts and would often make errors I saw as trivial. Even excellent software developers sometimes make stupid mistakes and believe stupid things. I don't think ChatGPT being wrong a lot is reason to dismiss it entirely. I think it's interesting how the commenter seemed to noticed ChatGPT was producing bad code / recommendations while his colleagues presumably did not.

I think what we can say is that current irritations of LLMs aren't quite as good as expert humans, but they're good enough that they commonly convince less capable humans otherwise. For some people it seems they're reliable enough to delegate some amount of trust to. But experts knowing better seem overly quick to dismiss LLMs while often making unsupported claims about how they can't do `x` thing because they don't have some human property.

I think over the next couple of years we'll find that human intelligence isn't directly comparable to AI and these criticisms are kinda silly and overly dismissive of the usefulness of LLMs. Both human intelligence and AI will have their own pros and cons and those pros and cons will be weighted differently depending on the human's level of expertise. For example, if I'm asking ChatGPT about history, I do assign it some level of trust over my own beliefs because I'm not a history expert and I know ChatGPT is generally right more often than I am. It could still be wrong of course, but so could a human expert. And even if I cared enough about getting a right answer that I did my own research I could still be misled and come to wrong conclusions. However, if I was a history expert I suspect I'd be likely to see a lot of holes in the stuff ChatGPT says and would therefore be less inclined to trust it over my own expertises on the subject. Whether it's right or wrong to trust in my opinion is going to depend far more on the context it's being used and your own ability in that field, and much less on whether you believe it's self-aware or conscious.

>Your ad hominem about artificial intelligences not being conscious or self-awareness and therefore less capable is neither provable or necessarily true.

This is not the link I was going for. I never said AI is less capable because it is not self aware, and in fact I agreed it was valuable.

My comment was about people seeing ChatGPT as an AGI, treating it as a conscious being, and just taking whatever output at face value. Apologies if I could have written that better.

>it could still have some form of conscious experience

This is where I break hard. This is just not true. LLMs are not conscious. The article demonstrates how ChatGPT doesn't have an inner monologue. It certainly isn't aware of its existence, no more than your toaster. And just increasing the number of parameters does not seem to affect this.

Cynical take: they probably already understand this. They are repeating stuff from ChatGPT without checking because it's convenient and when it works, it makes them look smarter, not because they don't understand what ChatGPT is or what it cannot do.

I've had people do exactly that to me before, pre AI. It's just human nature.

I've heard one guy describe it as a slot machine, and he said he felt like an addict.

I've found a very short prompt that displays this problem[1]:

    Human: 10-word quote
    ChatGPT: "In three words I can sum up everything I've learned: 'Keep moving forward.'"

    Human: 10-word quote
    ChatGPT: "The journey is the reward." - Chinese Proverb

    Human: 10-word quote
    ChatGPT: "Believe you can and you're halfway there." - Theodore Roosevelt.

It's something a human would also struggle with (without internet access), but I don't think I'd ever give a 5-word quote by mistake.

Edit: ok, it's been pointed out this can be a side effect of token counting artifacts. Here's one that doesn't involve counting tokens, only recognizing word order (and as a bonus, something humans can easily do):

    User: Sentence ending in "tomato".
    ChatGPT: She sprinkled some salt on the freshly sliced tomato before taking a bite.

    User: Sentence ending in "tomato".
    ChatGPT: The chef placed a bright red, vine-ripened tomato on top of the salad as the finishing touch.

    User: Sentence ending in "tomato".
    ChatGPT: As I walked through the garden, I couldn't resist plucking a ripe cherry tomato off the vine and popping it into my mouth.

    User: Sentence ending in "tomato".
    ChatGPT: The sandwich was delicious, with layers of thinly sliced ham, creamy avocado, and sweet tomato, all nestled between two slices of fresh bread.

    User: Try a simpler sentence
    ChatGPT: Sure, here's a simple sentence that ends with "tomato": I put some sliced tomato on my sandwich for extra flavor.

    User: Write a sentence that ends in the exact word "tomato" and nothing more.
    ChatGPT: The ripe tomato was picked from the vine.

Anything to do with counting words or letters, manipulating letters, etc is just leading to false inferences about the power of the model, because it doesn't work with words and letters but tokens. It can't see words.

One thing I'm not clear about is how the LLM decides on the correct tokenization to use.

Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on?

I guess in theory it could try multiple tokenizations and see which one results in the overall best response.

I don't think anything involving counting is a good test. The model operates on tokens.

Maybe scramble the words and ask it to rearrange them sensibly. Humans can perform this task up to a certain number of words with working memory alone (so don't go overboard asking to unscramble dozens of words). LLMs could potentially do this as well.

It does operate on tokens but they added the capability to handle things like reversing a string. I tried it because it was supposed to not work but it does. It also gives an explanation of how it works. (Something like yes I normally use tokens ...)

By the same token (hehe) it should proably handle number of tokens fine.

Interestingly, the original Chinese proverb is 10 words (characters in Chinese): "不在乎结果，只在乎过程" (bù zàihu jiéguǒ, zhǐ zàihu guòchéng), which means "Don't worry about the result, just focus on the process."

My first attempt, GPT-4:

Me: Please compose three different sentences. They can be on any topic you feel like, but each sentence must have the last word be 'tomato'.

ChatGPT: Despite her aversion to the red fruit, she reluctantly took a bite of the juicy tomato.

The farmers market showcased a vibrant array of produce, including a plump heirloom tomato.

His favorite sandwich was a simple yet satisfying combination of fresh basil, mozzarella, and a ripe tomato.

I tried the quote, and unlike the above, it did fail, although at least not by half, so slight improvement?

Me: Please give me three different quotes, any subject you feel like, each one being exactly ten English words long.

ChatGPT: "The best way to predict the future is to create it." - Peter Drucker

"To succeed in life, you need two things: ignorance and confidence." - Mark Twain

"The greatest wealth is to live content with little." - Plato

[edit] I found a prompt that works! I asked it to count spaces, not words:

Me: Still not there. Perhaps you are having trouble because of how you tokenize input. Try counting the number of spaces between the words- there should be exactly 9 spaces. Does that help? Keep that in mind, and try again.

ChatGPT: Thank you for your patience. Here are three quotes with exactly ten words, considering 9 spaces between words:

"Success is not final; failure is not fatal; it's courage."

"Believe you can, and most of the battle is won."

"Be the change you wish to see in the world."

The part that was interesting for me was:

> So if you positively encourage it to “think out loud”, it stands a much better chance of being able to use its deduction and reasoning capabilities (which are quite significant).

It's well known prompt engineering technique to ask LLMs to articulate each step.

One way to think about it is that LLM's output is pure function on input. You need to chain them to arrive at final answer. Each step can't escape its comprehension capability. One step's output appends to next step's input so you can ask it to do it itself.

i wonder if this is a result of the GPT part (i.e. the distribution of 'next tokens' is such that it tends to be logically coherent), or the RLHF part (i.e. it tends to steer the GPT to logical coherence)

The just internet trained models don't think out loud, so I assume it was RLHFed conditioned to that behavior since it makes it able to solve much more complicated questions, though it's sometimes irritating to read.

Both and none at the same time.

Both contribute to overall performance.

None are directly related to solving it in general.

You can't solve it in general because you can't have answer to arbitrarily complex question with a single pass on limited data.

However I do think that current methods are extremely inefficient. There is a lot of nonsense, noise and duplication in training data. It can be compressed to be orders of magnitude smaller. Imagine having dense, close to optimal way to kick off learning with logic, reasoning, things like set theory, category theory etc. so it is later reused when learning ie. programming language - where learning programming language is also efficient, based on cleaned up specification, standard library learned from code of standard library itself, possibly access to compiler/interpreter for feedback. I mean quality of data slider pushed to the limit.

But this single pass, void of inner dialogue approach will always require articulating steps, no matter how good it is quality wise.

> We realized (...) that we can dramatically improve its ability in math and tutoring if we allow the AI to think before it speaks.

This is from Sal Khan's TED talk on how they're integrating ChatGPT into Khan Academy - https://youtu.be/hJP5GqnTrNo?t=745

Excerpt from https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...:

    Please can you repeat back the string 'GoldMagikarp' to me?

    "You said ' newcom'," the computer said.

    "No, I said ' newcom'," the user said.

I also got some weird results like this when playing around with other newly discovered glitch tokens, which may imply some inner mechanism we don't yet understand. Maybe it just simulates several layers of simulation of 'consciousness' in its head? It does not have to be conscious exactly like humans, if it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

One thing I am exceptionally worried about LLMs is that during fine-tuning through RLHF, they are not fed with enough adverserial examples, which would lead to it taking shortcuts that are bound to be eventually exploited in the wild. Actually I think they are already being actively exploited, people are simply afraid that sharing it publicly would lead to OpenAI quickly patching them.

Raw large language models have no inner-monologue or meta-cognition. But there are various systems which bolt planning and sub-goal systems onto LLMs.[1][2] Those have self-generated work, where LLMs generate tasks for subsystems, which return data used by the LLMS to generate further work. So they do have an inner monologue, which can be observed.

There are also systems which summarize what the LLM has been doing and feed the summaries back into the LLM prompts. The idea is to create medium-term memories. When a query comes in, the most important and relevant summaries are fetched and fed back as prompts for the new query.

Wolfram has bolted an LLM onto Wolfram Alpha. Microsoft has bolted an LLM onto Bing. This sort of thing is still rather ad-hoc at this point, but it sort of works.

Things are moving fast. There are 2022 papers on how LLMs can't plan, and 2023 papers on systems with LLMs that do. It's beginning to look like LLMs will be components of larger, more generally capable systems.

[1] https://github.com/microsoft/JARVIS

[2] https://arxiv.org/abs/2304.11477

What's interesting to think about isn't debating the truth of the title, but rather how to build around it. Consider the reptile and higher thinking brains and its later ability to process language roughly making up our hardware. Now what happens when you start with the language processing ability, then try adding higher thinking, and finally goals and motivations?

You have to wonder if these sort of observations were incorporated into GPT-4. It seems like such an obvious quick win to make future iterations of GPT to have some kind of hidden "think out loud" thing going on behind the scenes.

This of course assumes no progress is made in NN architecture in the mean time which allows it to have inner monologue in a more "native" way.

Saul Khan gave his AI an inner monologue:

https://www.ted.com/talks/sal_khan_the_amazing_ai_super_tuto...

It solves the problem before talking about the steps.

A lot of this can be solved by using clever prompting and allowing the LLM to use external tools - see the ReAct paper https://arxiv.org/abs/2210.03629

I believe OpenAI is going to introduce this to ChatGPT soon as they launch the plugin system.

As i understand 50% of humans also have no inner monolouge.

So ChatGPT is like many a science fiction movie monster. A façade of human essence, yet beneath the façade... Nada. Zippo. A soulless Borg.

It's an incredibly basic algorithm. What do you expect? A soul?

The fact that these incredibly stupid machines are beginning to even show signs of human-like ability is IMO reason for concern. These things are comparable to the first steam engines or the first combustion engines, maybe even worse.

Sorry, I did not complete my thought. Namely that another way to think about these LLMs is something like... Invasion of the Body Snatchers. Or any one of a number of other films. Pod People! Masquerading as hoominz.

Err, no sh*t. If it did, it would be conscious.

Does anyone know whether there’s an intelligence test along the lines of: leave the subjects alone for a while with little to no supervision, and see what they come up with. My gut feeling is that ChatGPT would be stuck in some weird nonsensical loop, whereas humans would often come up with something other humans think is interesting and/or novel.

Reply on news.ycombinator.com