> From the script it appears that GPT4 is fully 'aware' that the FTP server is not real
Is "this word fits here" now synonymous to "fully aware"?
Is "this word fits here" now synonymous to "fully aware"?
The way it feels to me is that I have a fully formed concept almost immediately upon hearing or reading something then I employ language to try and convey the same concept into to someone else’s brain.
During that process additional ideas may occur to me, and that may augment the final output.
I experience some of what you describe, too, when dealing with ideas I already am familiar with, or explaining them to others. But when I'm trying to work through some novel thoughts, or am engaging in a conversation - including the one that has me writing this comment - I function in the way that's peculiarly similar to LLMs. That is, what pops into my head are words, phrases, sometimes entire sentences (often movie or videogame quotes). I pick one that fits best as an extension of what was already said - the previous words, the previous sentences, the response from the other party - and then repeat with the next completion, and the next, etc., until I feel it's time to stop.
I've joked for many years that, based on my personal experience and introspection, human brains at the highest level may just be glorified Markov chains. I sometimes still say that about LLMs - but I do so more as a compliment now. GPT-3 and GPT-4 may be just fancy autocomplete, but so are we. I feel we've brute-forced our way into a turning point in our understanding of what intelligence is and how it works.
Like I read your comment. I had thoughts during that process, but I read until the end, then I made a decision about what I thought, and that decision contained all the information that I want to convey in my response.
Now: when I am constructing my response, I am of course modulating which words might come next that would best fit into my language model to convey that concept, but the concept is an instantaneous understanding.
That instantaneous understanding may change during the process of articulation, but not fundamentally -- I think that any changes during the process of articulation could be considered "elaboration" rather than a reconceptualisation.
All the language that I just typed was designed to convey a concept that was more or less instantaneously formed and fully complete prior to my typing anything. The way in which I construct the language to articulate that concept may be very similar to the way in which a LLM constructs its responses, but I think the core difference is that original and instantaneous conceptual understanding of what it is trying to articulate.
Only moments before I write the word, do I know what that word will be, but I know the complete thought I want to convey. I don’t know if this is important, but my thoughts are not words, so I’m basically just describing them using words.
The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression is like an instantaneous graphical image of an elephant, that one must then put into words.
I think that a LLM doesn't have that. I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language. The concepts are all pre-existing in the same way that all the chess move possibilities pre-exist in the programming of a chess computer.
EDIT: In fact even the way I described concepts as "pre-existing" there might be giving it too much credit. Like the concept only arises on the receiving end, from a pool of concepts that were pre-existing but about which the LLM has no "awareness" whatsoever.
The question is, do you know its attributes before you describe them to yourself? To me, at least, an "elephant" feels like a pointer, a handle, a promise - I know it's a thing, but to know more than that, I have to dereference the pointer. And that usually means something that's getting close to stringing words together.
Possible confounder: I suffer from aphantasia[0]. I have nearly zero ability to perceive images in my mind, except when dreaming. Maybe it's easier to "know immediately what it [the elephant] is and all it's attributes", if it just pops into your head as a complete image. I, unfortunately, don't experience that.
> The difference of course is that the concepts can change and merge with each other, but what I'm referring to as an "instantaneous concept" that precedes linguistic expression (...)
> I think that a LLM doesn't have that.
Crazy idea: what if a concept is... a partial set of coordinates in the latent space? Enough to describe a pretty narrow region.
> I remember Kasparov had some criticism of Deep Blue that it wasn't really "playing chess" (although I can't find it now). In the same way I think that LLMs aren't really conveying a concept, they're only doing the part that assembles the language.
My tentative guess is that language will turn out to not be the fundamental part of what makes LLMs so powerful/spectacular. I think the magic is the latent space itself - the high-dimensional constructs where all kinds of associations end up represented as spatial distance. Language is just a serialization format[1] for latent space associations. By being fed lots and lots of text in training, LLMs build up the latent space, piece by piece, until there's enough dimensions to it that it is no longer a simple mapping between words learned from the training data. I think this is how understanding and concepts look like.
Also, I think it takes so much input text for LLMs to get to their current performance, because LLMs learn like babies - they soak up everything they can, letting the inputs self-arrange by correlations. We learn to understand what an elephant is by hearing stories about it, seeing it alive, and/or being told facts about it in a structured way (i.e. school). LLMs learn what elephant is by soaking up so much text that (the token representing) the word "elephant" is, in the latent space, close to all the words people used to describe its attributes - which themselves are close to other words with whatever various meanings and aspects we relate to those attributes, etc.
EDIT:
And connecting the two parts of the comment: LLMs are AI's poster child of aphantasia. If they have concepts, how else could they work with them if not by... articulating them? And when they articulate a concept, what makes us so sure they're just parroting associations, and not truly having a concept?
--
[0] - https://en.wikipedia.org/wiki/Aphantasia.
[1] - Quite literally, as communicating via any language happens via ingesting or producing a series of some kind of tokens (letters, words, ideographs, etc.) over time.
I recognize that mode. I have it to. I had it now; this comment is aiming to transmit a thought that came to me fully formed while reading your response. That said, it's a bit suspect how it popped up as if my mind was doing an associative search. It's also interesting that, however fully formed the thought might have been, it's also small, as if a conceptual token, and I usually have to run further associative lookups to extend it before the comment (or train of thought) feels satisfactory to me.
In the previous paragraph, the last two sentences were not part of expressing the fully-formed thought described earlier. I'm not lying to you, it's just I didn't expect to write the second half of the previous paragraph (and this paragraph) when I was writing the first one; they came from conceptual autocomplete.
> That instantaneous understanding may change during the process of articulation, but not fundamentally
For me, it quite often does. I rely on it happening often. Example cases:
- On HN, I post maybe 1/3 of the comments I write. 2/3 of the time, the act of trying to express my thoughts makes me realize that a) I'm wrong, and/or b) it's a bad idea to continue (e.g. I'm just procrastinating, or posting something impolite, or not useful, etc.). Of the 1/3 that survive long enough to be posted, I still delete some - as sometimes the realization from the previous sentence hits me only a couple seconds after posting.
- When facing difficult problems, I do a lot of internal monologue. In it, exploring whether certain phrases feel right, and kind of sculpting the chain of thought, is a critical part of the process.
- Because I often feel I lack the working memory to keep enough thoughts in flight, I often "talk" with myself via a text file - externalizing my internal monologue. That process also involves sculpting sentences, paragraphs or bullet points until they feel right, so it's arguably still articulation. And the more I think about it, the more this looks to me like the kind of tricks you do to work around the token limit / window of GPT-3 / GPT-4...
EDIT:
Ultimately, I think we're both doing thinking in similar ways, perhaps with different proportions between modes, and different conception of it.
One relevant conceptualization I have that I remembered just now: in conversational situations, I often feel like my subconsciousness is a Markov chain / LLM autocomplete, spitting out phrases, and my consciousness is busy judging them, and choosing whether or not to pass them to the actuators (speech organs, or fingers on the keyboard).
The two "models" - the autocomplete and the censor - is something I can clearly see in my mind when I'm processing a problem in a calm manner, or participating in a slow-paced discussion. But I've noticed that the more excited I get (or intoxicated), the weaker the "censor" part is; in the limit (when I'm really excited, angry, or drunk), I feel as if the Markov chain bit is feeding phrases straight to my mouth/fingers, cutting out the middleman of consciousness.
(Of course, what I wrote is all "how the algorithm feels from inside", so it's not very objective or reliable...)
Or at least I used to, because I only recently realized that, over the last few years, I've mostly switched to thinking in English only. I'm now in a weird state: my native language (Polish) is still easier for me to use, but also using it feels wrong somehow. Often enough, when talking to people around me, I notice I'm translating my own thoughts from English.
It might be that this process of learning and internalizing a second language, to the point it eventually took over my first one, is what changed my experience of the thinking process itself.
Ultimately, I think the definite answer would require solving the halting problem. The approximate answer we should be able to get when we let models interact with some complex environment over time. That is, the model predicts the next token, which may cause some change of a shared state, and then the model receives a response.
This seems like an answer to the question, "why is the LLM so compelling?" And not "what makes something conscious/aware?" Which is an important question but feels like not quite the right one to ask and then come to this conclusion. And it bothers me if I try to accept an idea where I don't know what conditions I'd reject it (eg, I don't know what evidence would cause me to reject this hypothesis, I can't see a test I can perform to next reject it).
ChatGPT 'predicts the next token' so much better than other 'next token predictors' that using that as the comparison for a low effort dismissal is like saying "what's to stop a human tracking a criminal by scent, humans have noses and dogs have noses so they must be the same".
If we agree that Markov chains aren't aware, then surely we also agree this definition of what makes them aware is either incomplete or incorrect.
Predicting the next token turns out to be quite profound, I'm just not convinced, at this time, it's sufficient to be aware of conscious. If you have a reason I should believe it, I'm all ears.
If you want to test how good a chatbot is at predicting tokens, well, you just have to test it. You can throw a Winograd schema challenge at it, or let it take part in code competition or exams, let it solve puzzles and all that. Quick simple test will for example show you that ChatGPT isn't any good at calculating, as the answers will just be wrong.
As for awareness, we know what ChatGPT is aware of: Its 4000 token history and whatever information it learned from the training data. That's the grand total of its existence, it doesn't get any other input, it doesn't perceive time, it doesn't learn, that history is all there is to it. And we can call that awareness as ChatGPT is able to perform correct prediction based on that token history.
For consciousness there are still a lot of basic parts missing, such as the perception of time, senses to act with some external world, the ability to think for itself without getting explicitly prompted and all that.
I think you and I are generally in agreement about consciousness and awareness. I think consciousness is an interaction of an agent with it's environment, rather than an agent in isolation. I guess that's why I find this attitude that consciousness is a matter of feeding enough data into a static training process confusing. I think it's a kind of is-ought problem where I ask about the "ought" and people express the "is" and we end up talking past each other.
But I don't really have a good hypothesis to disprove this either, so I'll have to meditate on that. I think it may be testable but it's challenging to model.
I wouldn't buy it, but you can sell it to plenty of people.
You even ignored the quotes that OP put around ‘aware’.
As someone working with GPT-3 day-to-day at $dayjob, these words - usually reserved for sentient beings - are simply put typically the most apt to use in pretty much all contexts other than some sections of an academic paper. The fact that you yourself couldn’t get the sentence to work right is a testament to that.
Again, as someone that’s working with and having conversations about this day-to-day, we all find ourselves using these words, first kind of awkwardly, with laughs and air quotes, and we try to mentally reach for the best word to use. Eventually we all just know what we mean, and that’s that.
So, if I can ask directly, how do I talk about this online in a way that allows for productive conversations about the very obviously disruptive abilities of LLMs but that still avoids your snark?
And like someone said before, perhaps we can think of the GPT models not so much as “artificial intelligence”, but more like a different kind of intelligence from what humans have. With strengths and weaknesses of its own.