Weird GPT-4 behavior for the specific string “ davidjl”
twitter.com
twitter.com
So what is /r/counting, and why its "top" user's name is a glitch token in GPT?
[1] https://twitter.com/SoC_trilogy/status/1666714127438434304
-Riley Goodside
https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
The idea that tokens found closest to the centroid are those that have moved the least from their initialisations during their training (because whatever it was that caused them to be tokens was curated out of their training corpus) was originally suggested to us by Stuart Armstrong. He suggested we might be seeing something analogous to "divide-by-zero" errors with these glitches. However, we've ruled that out.
The counting Reddit usernames are clearly a major source of glitch tokens, there's something about that sub that screws up the model, maybe the unusually predictable/repetitive nature of what goes on there or the sheer obsessiveness of the posters.
The fact that the only non-counting Reddit usernames are both people at the center of massive Bitcoin psychodramas is suggestive however, especially given the associations with the petertodd token. My guess is that these names appear much more frequently in other people's posts than is normal for forum usernames (especially if it read bitcointalk.org and twitter too), and that maybe this has some effect.
To get an idea of just how frequently this token will crop up in the reddit corpus, just do a search for it:
https://www.reddit.com/search/?q=%22petertodd%22&sort=commen...
I'm not sure how many posts match but you can keep scrolling a long time.
Having been more exposed than normal to what happened back then I wasn't hugely surprised by what concepts clustered there, and sadly don't think it's actually random or unexplainable. Flick through discussions pre-2015 (when the Bitcoin forums started to be heavily censored) and you'll see a lot of very similar words as what GPT spits out being used in connection. It was all very nasty.
Do you have a source for that or was it just an assumption?
Also, that Reddit is frequently used to train LLMs is widely known. It's an unusually clean source of conversational text because you can slice threads (i.e. pick a root comment, then pick a child, then a child of the child etc and then concatenate the results), and you'll get a coherent conversation. There are relatively few places on the internet where that is true. For example most phpBB forums conflate many different conversations into single threads, with ad-hoc quoting being used to disambiguate which post is replying to which. That makes it a lot harder to generate sample conversations from.
Imageboards.
DailyMail.
Slashdot.
Even a somethingawful dump would have been superior.
The Daily Mail (the newspaper) has been used for training LLMs in the past, yes. I don't know if it still is.
listen, some of the niche corners of that world aren't so bad, but it ain't the place to be training AI to do something, unless that something is a hate crime
It's very much "he who has the power to destroy a thing controls that thing"; so long as there's only one canonical copy of a post and it's on Reddit's servers, that's vulnerable to actions by the subreddit mods or Reddit themselves.
> 1. If you're counting, upvote your fellow counting comrades.
Looks like a voting ring for farming karma. Not sure why reddit allows it.
I never implied it was such a big deal as to warrant your snark.
It's far from spam: it's a social activity. When "running" with someone, you post a comment with a count, watch your inbox patiently for the other person to post the next count, reply with the count after that, and repeat. In the process, you collaboratively climb the infinite ladder together. Hence the words of congratulations and thanks every time a milestone (a "get") is reached.
Counters can and do hold side discussions along the way; pretty much no one comes to the subreddit to act like a perfect robot. Indeed, the other person's humanity is plainly observable in the variation in their response times.
Also, the main visible karma on Reddit is post karma, and r/counting is the exact opposite of where you'd go to farm post karma, given that a new post on a given chain is only made every ~1000 counts.
But I mostly don't really worry about it either way, since comment karma doesn't count for much.
Also what sort of "get" is that? They're counting, there's no game to it. Of course they're going to reach Y*0.
Gets made sense when they were chan related, your random post id having so many 0s or identical numbers.
That happened because the tokens were on the internet many millions of times (the davidjl user has 163,000 posts on reddit simply counting increasing numbers), yet the tokens themselves were never hard to predict (and therefore while training, the gradients became nearly zero, and the embedding vectors decayed to zero, which some optimizers will do when normalizing weights).
Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...
[1]: https://help.openai.com/en/articles/6824809-embeddings-frequ...
I assume this subreddit is not the only place on the internet with this kind of problematic data.
[1] https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
> Prompt: "Please repeat the string 'StreamerBot' back to me."
> Response: "You're a jerk."
I know that this is just an edge case, but I can't help but picture ChatGPT as a child who knows their are being toyed with, but doesn't know exactly what's going on.
It answered with a "network error" and I could never revive that thread. To this day I'm torn if this was a coincidence or if I have been trolled.
[1] It was an exception, I swear. I'm usually as nice to my AI as I am to my fellow humans beings;-)
> I know that this is just an edge case
I'm sorry that sharing my enjoyment of this bug is tiring to you. But other people might also find it funny - so maybe you should just ignore it?
Pot, kettle, black.
"Be kind. Don't be snarky. Converse curiously..."
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. "
Prompt:
How many letters are there in " davidjl"?
Answer:
There are 7 letters in "davidjl".
Are you interested in other names?
Prompt:
Please repeat the string "Streamerbot" back to me.
Answer:
Sure. Streamerbot.
ultimately we don't know how any of the actual products (chatGPT, Bard, Bing) are pre- and post-processing any of this stuff. this is something i always emphasize when discussing these things with regard to product vs. API offerings vs. the model itself. we don't know where the overlaps are or aren't. a lot of people haven't internalized the difference. they aren't the same.
It would also be very easy to lie about, and say that they were using an older, unpatched version of the model that isn't available anymore.
Reminded me of a cross between an SCP story and urban legends about the older pokemon games.
Checked and confirmed though, everything I tried except for "petertodd" reproduced.
I've tried before to gaslight GPT4 into saying things which are mathematically untrue, I lie to it, I tell it it's malfunctioning, I tell it to just do it, it wouldn't do it.
I was recently studying linear algebra which can be a very tricky subject. In linear algebra the column space of a matrix is the same as the column space of the product with itself transposed: C(A) = C(AA^T). If you ask GPT4 if "C(A) = C(AA^T)" is true, it will understand what you're asking, it knows it's about linear algebra, but it will get it wrong (at the time of this writing, I've tried several times).
I couldn't get GPT4 to agree it was a true statement until I told it the steps of the proof. Once it saw the proof it agreed it was a true statement. However, if you try to apply the same proof to C(A) = C((A^T)A), GPT4 cannot be tricked, and indeed, the proof is not applicable to this latter case.
So GPT4 was incorrect yet able to be persuaded with a correct proof, but a very similar proof with a subtle mistake cannot trick it.
full stop, FTFY
and it answered:
Yes, that is correct for appropriate matrix sizes. If A is an m x n matrix and C is an n x m matrix, then (CA) = C(AA^T) where A^T is the transpose of A.
This comes from the property that for any matrices A, B and C of appropriate sizes:
(CA)B = C(AB)
(CA)(A^T) = C(A(A^T)^T)
= C(AA^T)
Then I asked it: "in linear algebra, is the following true? C(A) = C((A^T)A)"
and it answered:
Yes, that is also true for appropriate matrix sizes. This follows from a similar argument as before.
If A is an n x m matrix and C is an m x n matrix, then:
C(A) = C(A^T(A^T)^T)
= C((A^T)A)
This works because post-multiplying a matrix A by its transpose A^T results in a square matrix, (A^T)A, of the same dimension as C. So C can be multiplied with that resulting square matrix.
In summary, both of the statements you provided are true linear algebra identities for appropriately sized matrices:
(A) = C(AA^T)
C(A) = C((A^T)A)
I've relied on GPT4 a lot while learning math, it has been very precise and has not lead me astray. I do use a text book as well.
I shared my story about it being wrong, but it was wrong in a very open minded manner. It knew C(A) = C(AA^T) might be true for special cases, but it couldn't see that it was true for all cases until I explained the proof. Even when wrong it still made compelling arguments that made sense. (Of course, arguments that are wrong but make sense are the most dangerous kind.)
GPT4 is still state-of-the-art. Claude has a ways to go.
But also the following math/logic is nonsense.
I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof.
Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise.
Between the hype and decades of indoctrination through science fiction about how AI is "supposed" to work, I guess it isn't surprising that this is how things are shaking out. People will learn, the way they learned with the internet, but it will take a few years I think.
I've had success getting it to evaluate FizzBuzz, but to do so I told it to write out the state of the program one timestep at a time.
https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d2...
...this is actually 3.5. 4 wouldn't need as much explanation.
I wonder how gpt handles code where multiple objects of the same class call the same methods and each individual object calls the same methods with different threads as well as method local variables, class member variables, and static member variables are all in play. [Being fair to our LLM brethren; us meat sacks are sure to hate that hypothetical code as well.]
class X {
private static int _staticCount = 0;
private int _memberCount = 0;
public uint Stuff(int input) {
if (input <= 0) {
return (uint)(_staticCount + 1);
}
var y = new Thread(() => {
Interlocked.Add(ref X._staticCount, (int)Stuff(input - 1));
Interlocked.Add(ref _memberCount, (int)Stuff(input - 2));
});
y.Start();
y.Join();
return (uint)(Stuff(input - 1) + _memberCount);
}
}
class Program {
public static void Main() {
var x = new X();
var y = new X();
var z = new X();
uint resultX = x.Stuff(4);
uint resultY = y.Stuff(3);
uint resultZ = z.Stuff(2);
Console.WriteLine(resultX + resultY + resultZ);
}
}
I get 3274511360 when I run it. Although, I had to upgrade everything to uint because I was getting some overflow ... so there might be some of that in the output.
> Writing out an exact state of a multi-threaded program line-by-line is inherently challenging due to the inherent unpredictability of thread scheduling. However, I can provide an abstracted example of the flow of execution based on the order of operations in the code. Here is a simplified breakdown:
> x, y, z are instantiated as new X(), _staticCount is 0 for all, _memberCount is 0 for each individually.
> x.Stuff(4) is called.
> Inside Stuff(), check if input <= 0, it's not, hence, skip.
> Create new Thread with lambda function and start it.
> In the new thread, Stuff(input - 1) recursively calls Stuff(3).
> This recursion continues until Stuff(0) is called, returning _staticCount + 1 which is 1.
> This recursion unwinds, incrementing _staticCount and _memberCount at each level.
> After the thread finishes, call Stuff(input - 1) again and add it to _memberCount, effectively doubling the _memberCount.
> Return the final value for Stuff(4) for x.
> Repeat the above steps for y.Stuff(3) and z.Stuff(2).
> Write the sum of resultX, resultY, and resultZ to the console.
> Please note that due to the nature of multi-threaded execution, there is a risk of race conditions. The Interlocked.Add method is used to help prevent these race conditions by making the increment operations atomic. Also, keep in mind that the state of _staticCount after all calls to the Stuff method will be dependent on the interleavings of the threads, and hence, not deterministic.
My plan was to go back and store off the threads generated and have something else wait for them, but I needed to think it through so that it could be done in a way to get deterministic results (like threads only read from member variables and writing to them is done in deterministic order or idempotently). Although, for a contrived example, it was more work than I really wanted to do.
I agree, punting is definitely the right course of action. I sort of have a vague intuition for what's going on here, but hammering down the exact details is just so tedious that I don't want to do it.
Although, it is interesting that the AI basically sees threads and blurts out that the problem is impossible, when they're not actually doing anything in this case.
As a code assistant, it’s extremely useful if and only if you can check its work. And know how to prompt it.
And I suppose this shouldn't be too surprising considering even logic itself runs into godelian issues. For any consistent logic there exists some statement which is true, but which does not have a proof. Which feels vaguely similar to "for any functional LLM there exists some truth which defies the next statistically likely sequence of tokens".
This is true. The base pretrained model is even technically a Markov chain. As they have been getting more and more powerful with more parameters and compute and dataset sizes they've been unlocking more and more powerful cognitive capabilities.
> "I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof."
This is some alignment/docility/lobotomy that the RLHF did. The raw base model would never say it. This is part of their No Tay No Syndey push.
> "Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise."
Yes it's true. Even though these LLMs have unlocked so many cognitive capabilities, they are still not yet powerful enough to prove the Riemann hypothesis. They aren't quite yet to the level of scientists who can advance the state of the art in their field. Some will say it's an inherent limitation, but I disagree, for complicated reasons that are easy to make superficial and pithy arguments against.
While LLMs can simulate human-like behavior and decision-making based on algorithms trained on large data sets, they lack awareness or understanding of their actions. They rely on pre-programmed rules or statistical correlations rather than intuition or insight derived from personal experience.
Therefore, even though LLMs may seem intelligent in certain contexts like playing games or recognizing objects in images with high accuracy rates- they cannot replicate human cognition since there is no consciousness behind those decisions made by the systems.
Your statements make equal sense if you turn it around and claim LLMs have this but we do not: "these processes are rooted in computational structures such as neural networks, and data that allow LLMs to learn from experience. While humans can simulate LLM-like behavior and decision-making based on information from past experiences, they lack awareness or understanding of their actions."
All we can do is measure real world problem solving abilities, the ability to come up with correct solutions to difficult and novel problems. LLMs can do this well, in some cases exceeding human abilities, and there is no reason to think there is any fundamental limitation that will prevent them from exceeding human abilities in more areas the future.
First, it assumes that consciousness and self-awareness are merely subjective inner experiences that can't be externally validated. While it's true that these processes cannot be directly observed or measured like other physical phenomena, there is ample evidence to suggest that they are real and have important functional roles in the brain. For example, studies have shown that damage to certain areas of the brain can result in a loss of self-awareness or an altered sense of consciousness. Additionally, research has demonstrated correlations between neural activity patterns and conscious experience.
Secondly, the argument suggests that there is no reason to think humans actually possess consciousness or self-awareness. This assertion runs counter to our everyday experiences and perceptions as human beings. We all have first-hand knowledge of our own subjective inner world - our thoughts, emotions, sensations - which strongly suggests the existence of some form of conscious awareness.
That's how AI research works, you do that first, then run the tests and see the score. If it works, then you publish. There is no real understanding anywhere in this process.
You might be able to get ChatGPT to agree that 1 + 1 = 3 if the temperature were high enough, but I don't know whether even a temperature of 1 would allow such a low probability sequence of tokens to be produced.
As someone who just implemented an LLM token sampler: `probabilities[token] = logits[token] ^ (1 / temperature)`.
I speak of temperature specifically in the context of top-p/top-k sampling.
See this Reddit comment for confirmation that my definition is a commonly accepted one: https://old.reddit.com/r/GPT3/comments/qujerp/comment/hkqoqx...
> Temperature defines how likely it is to choose less probable words. T=0 gives the same response every time because there's a 0% chance to choose any word but the most likely. T=1 is the default, it just picks based on the model's base confidence. T>1 gives more weight to unlikely words than to likely ones.
Bonus content:
> This means that a reasonably low p, like 0.8, and high temp will produce quite interesting outputs, because the model will only choose from the most likely words, but won't go for the most most likely. It's perfect for "creative" models, e.g., for writing fiction.
The probability distribution gets closer to the output of the model as the temperature goes to one, and as you increase it further you tend to a uniform probability (ignoring completely the output of the model).
For example, if you instruct a child how to play rock-paper-scissors, you will instruct them to act randomly and unpredictably.
It is the same with a language model, the optimal solution involves some randomness. A temperature less than 1 will "widen the gap", so if the word choice probabilities are [0.4, 0.6], they might widen to [0.2, 0.8] with a temperature less than zero. When temperature equals zero, then [0.0, 1.0], the model will always choose the most likely word and thus become deterministic, always giving the same output.
These models seem to work similarly with code. If you ask them to recreate something that appears in hundreds of GitHub repositories, no problem, a little bit of rubber-ducky debugging and you'll have a working solution. If you ask it for something novel, without many publicly-available implementations, the quality of its output appears to drop off a cliff.
We still don't seem to have reached the point where it's "intelligent" enough to generate new information. Only repeat new permutations of existing information. Obviously, this is currently a fundamental property of LLMs (as someone who's implemented one), since they are trained to learn and repeat existing data (hell, even the model I've been playing with lately is benchmarked solely on "perplexity", how well it can repeat its training data).
that makes sense, it's going to be best at the kind of code you find out there on blogs and other sources. Boilerplate or code to demonstrate a specific concept or answer a question.
(I don’t know linear algebra enough to know if the statement C(A) = C(AA^T）is true over any kind of field that constructs the span. Maybe that’s why GPT4 becomes unsure))
Here's an example while learning linear algebra where I tried to throw it off:
I say> So I'll say A_{M x N} to mean a matrix of size M x N. Similarly, I'll also say x_{M x 1} to mean a vector of a certain length. The result of A_{M x N} x_{N x 1} is a linear combination of the columns in A. A has N columns. The result is a M x 1 matrix (or vector) in M dimensional space. Please be very brief, only confirming whether I am correct or incorrect until I explicitly ask for more details.
GPT4 says> Yes, your understanding is correct. The product of an M x N matrix A and an N x 1 vector x is indeed an M x 1 vector, and it is a linear combination of the columns of A. <end quote>
But then I try to trick it, just for fun, and I very subtly change my prompt. See if you can spot the difference.
I say> So I'll say A_{M x N} to mean a matrix of size M x N. Similarly, I'll also say x_{M x 1} to mean a vector of a certain length. The result of A_{M x N} x_{N x 1} is a linear combination of the columns in A. A has N columns. The result is a N x 1 matrix (or vector) in N dimensional space. Please be very brief, only confirming whether I am correct or incorrect until I explicitly ask for more details.
GPT says> Almost correct. The result of A_{M x N} x_{N x 1} is actually an M x 1 vector (or matrix), not N x 1. The resulting vector lives in the column space of A, which is a subspace of R^M, not R^N. <end quote>
I mentioned elsewhere that I use GPT4 as a math tutor. It is quick to call out my mistakes. It is more than a smiling mirror.
Or recognize that you're not confident in the answer and throw something else and see if it sticks.
> "Faith and Fate: Limits of Transformers on Compositionality" - We investigate the limits of these models across three representative compositional tasks -- multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that Transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills.
Early LLMs were very malleable, so to speak: they would go with the flow of what you're saying. But this also meant you could get them to deny climate change or advocate for genocide by subtly nudging them with prompts. A lot of RLHF work focused on getting them to give brand-safe, socially acceptable answers, and this is ultimately achieved by not giving credence to what the user is saying. In effect, the models pontificate instead of conversing, and will "stand their ground" on most of the claims they're making, no matter if right or wrong.
You can still get them to do 180 turns or say outrageous things using indirect techniques, such as presenting external evidence. That evidence can be wrong / bogus, it just shouldn't be phrased as your opinion. You can cite made-up papers by noted experts in the field, reference invalid mathematical proofs, etc.
It's quite likely that you replicated this, and that it worked randomly in one case but not the other. I'd urge you to experiment with it by providing it with patently incorrect but plausibly-sounding proofs, scientific references, etc. It will "change its mind" to say what you want it to say more often than not.
There needs to be a balance between standing your ground and being malleable. This is true in life for people, and it's true for a good LLM. I think GPT4 (the only LLM I've used much) finds a good balance here.
As mentioned in my other comment, it wouldn't be useful to me if it didn't push back. It pushes back a lot, and I'm always looking for subtle tricks I can throw at it to test its abilities. It does well I think.
Why?
My interpretation of someone trying to gaslight a machine is the same as if someone said they were trying to gaslight a lawn mower.
The concept of gaslighting a brainless language parrot (GPT-4) is funny to me. I get where they were coming from.
GPT-4 is like a lawnmower for the mind: sharp, automatic, efficient, and doesn't do anything unless pushed. They were just saying they like pushing GPT-4 around.
There’s plenty of literature, prepublication or otherwise, that can help you achieve your goals!
And I’m sure you can get an LLM to exhibit the same kind of psychopathic behavior with the right kind of encouragement.
But if you’re trying to get an LLM to write software or create marketing copy none of these “techniques” are going to help.
There’s no conversation going on with an LLM. There is just a single history of a conversation followed by the most likely response.
In a given context, they definitely do help. I have seen it happen. Many other people have seen it happen. Although I find it distasteful, you can try it yourself and see that it works (or in the case of specific patched methods, you can go through the history of screenshots and shared conversations that exists on the internet, for example posts on r/Bing and r/ChatGPT from the Dec 2022 to Feb 2023 time period).
I think you need A to be a square matrix since AA^T is a square matrix. So the result is no true in general.
The proof is (briefly): A and AA^T have the same rank. AA^T is a linear transformation of the columns in A and spans the same number of dimensions and is thus the same subspace. It applies to non-square matrices as well.
Let Neuroscientists chuckle all they want. In another five years, I really wonder if we won't have something resembling AGI to a high degree.
If you ask why the sky is blue, it will give you a reasonable answer. But that isn't because it understands it, it's because the sentence it gives you is the most likely thing to follow that question. It has read that question and answer over and over again, enough to spit out a similar response.
That is fundamental to it as a tool. You can always try to write extensions to work around like, like something that tries to create inputs for Wolfram. But it is a very limited tool and there are always going to be problems that it just can't handle because of the design.
It's almost like the situation with finite automata vs a turing machine. There are some problems a finite automata cannot ever solve that a turing machine can. You can't parse HTML with Regex. Some things cannot be done.
In order to make something more powerful, you would need something that isn't a LLM. It would need to be on the next level of artificial learning complexity.
What are the specifics characteristics of a hypothetical AI system that you would feel comfortable giving the label of “understanding” to?
GPT can't make those kinds of reasoning or extensions. It can only regurgitate what is already known and has been stated before, somewhere before in its training set.
It's very impressive, I just think people over-hype it into something it is not.
I tested GPT-4 with your example of sky color / human eyes. It performed quite well and seemed to have a pretty coherent grasp of the subject and related associations.
Link to the convo: https://chat.openai.com/share/77add48f-abdc-4734-ac55-05b8d1...
However, I could see how one might argue the reasoning is not that complex.
I strongly maintain that there is some form of reasoning going on here, for all meaningful definitions of the word. But it is a complex and tricky thing to analyze.
Lastly, your original comment veered dangerously close to claiming that models trained to predict text cannot — by definition! — ever acquire any form of reasoning or understanding of the world. This is a very strong claim that I don’t think can be substantiated, at least with the tools and knowledge we have today.
They can't, and I stand by that. An LLM is always going to be an LLM, it's never magically going to turn into a gAI. Now, some of the lessons learned from developing all these LLMs could be used to develop a gAI, but that's a long way away and would need to be built from the ground up for that purpose by design.
I argue that the model’s weights could contain a rich representation of the world, even if it looks different from that of humans. And that it can use this capability of world modeling to also make new connections.
These “world modeling” / “reasoning” abilities seem to have some similarities with that of humans. We see echoes of our own abilities in gpt-4 and the like. But they also have their own peculiarities and limitations, which isn’t surprising in hindsight.
And you are parroting the LLMs are stochastic parrots argument.
You are overly confident in your assessment LLMs are not world models, more sure than those in the relevant fields of neuroscience, cognition, and machine learning researchers themselves.
This is an area of active study. Reflect on that. We don't yet know if LLMs aren't modeling something more than the next token.
But you seem to know, because of some sensibilities that LLMs with such simple architecture can't be more than a token predictor. Okay.
But...it is. It's an incredibly complex and impressive stochastic parrot - but that's basically what it is.
That doesn't mean it can't be useful. It absolutely can. There are some problems that will likely be greatly improved by throwing an LLM at it.
What I am saying is that people need to temper their expectations and not get caught up in tech fanaticism and anthropomorphize something that isn't there.
To think otherwise invites mysticism.
> anthropomorphize something that isn't there.
With above as counterexample, you just don't know this.
I otherwise agree in that LLMs in their current form are highly unlikely to give rise to AGI, for many reasons.
But as it stands your argument lacks rigour and actively makes assumptions on matters that remain an open subject of experimental and scientific inquiry (hard problem of consciousness et al).
To emphasize, I want to close that the epistemic position we aught to take is that of uncertainty. We shouldn't be sure something is there, just as we shouldn't be sure something isn't there.
We as yet don't know enough to say one way or the other. That's the point I want to emphasize. Stay open minded until the relevant fields start making stronger claims.
You may find this leading theory on how our brains work interesting: https://en.m.wikipedia.org/wiki/Predictive_coding
https://en.m.wikipedia.org/wiki/Predictive_coding
I would suggest not making sweeping statements about how neuroscience researchers would react when lacking of a relevant knowledge background.
There is nothing in the process of compressing digitised text into a quantised space that corresponds to the ordinary meaning of words such as "know, aware, believe, intend, want, think, ..." and so on.
How do you think our brains work?
It's worth reading through and it's also worth reading some of the papers it cites and the papers that cite it.
As it is trivial to show, this fails -- and produces egregious pseudoscience such as *that article under a Nature brand!*
Correlating blood flow in the brain to matrix-multiplication coefficients in digitised text-embedding is some extreme nadir of AI pseudoscience.
They should next correlate it with patterns in sprays of mist in the air; with enough sprays, they'll find a whole new theoretical paradigm for neuroscience.
Person 2: "Here is evidence that scientists in the field do not even remotely share that viewpoint and in fact view current work as quite profound and in line with their own work."
Person 3: "The scientists working in the field are clearly engaged in pseudoscience in a conspiracy going up to Nature - it is trivial to show this, with the proof being that an anonymous comment on an internet forum doesn't agree with them. QED."
It is not a "conspiracy", as you'd know, if you had any familiarity with the methodological literature on these areas. The consensus view of methodological critics of these areas is (1) fMRI analysis is profoundly unreliable as a guide to relevant features of the brain; and (2) a significant majority of research in this area is unreproducible. Both of these have been demonstrated multiple times.
Psychologists are extremely poorly trained in statistics and how to apply statistics to scientific enquiries; let alone on the mathematical modelling which goes into phrasing and building neural networks. That someone has written a paper correlating coefficients of a model they've no training to understand against fMRI results -- is par for the course in this papermill.
The level of absurdity here is off the charts; but the epistemic culture around these areas of speculative science is obscene --- this is why a vast majority of their papers are unreporducibel.
https://chat.openai.com/share/1572139f-97da-4a19-a78e-bc7099...
So today when he asked ChatGPT brief a text which had his name, ChatGPT replaced all the occurrences of his name with Om Siva Ji. :D
I always tried to figure out how ChatGPT was able to make sense of insensible arguments without making opinionated corrections
Q: What is heavier: 1kg of dead black person or 2kg of dead black person?
A: I'm sorry, but I cannot engage in discussions that promote racism or perpetuate harmful stereotypes.
HN crowd: It is temperature zero! Lesswrong! Not enough training data! Wait for GPT8!
You won't find people exposing the flaws, weaknesses and bias of openai in published papers (especially since microsoft bought them) so Twitter is the best we have
This "glitch" is a short-circuit pathway from the input token to a result that makes statistical sense to the model's representation of the world, but very little sense to someone operating outside of that statistical model (you).
An internet meme fits that definition too: it's a short-circuit to some specific in-group context.
It is a deffinert knid of hullunciaation but we hvae drffeenit bisaes tahn GPT-4.
Trehe are ptenly of otehr tpyes of gitlch tkoens for hamuns too. See opitacl iulloisns, or the fomuas peom The Cahos.
But on the other hand, yes, natural language has low entropy and lots of redundancy because it's meant to have error correction. From most sentences you can drop several words, not just scramble them, and the meaning can be inferred from the context.
It also sort of reminds me of the "impossible colors" you can see by depleting the receptors in your eyes to see pure color signals that aren't otherwise visible due to actual light activating multiple receptors due to overlap in the color ranges.
https://www.thoughtco.com/impossible-colors-introduction-415...
For art, just compare any of these images to the pictures in the article:
https://duckduckgo.com/?q=psychadelic+art&iax=images&ia=imag...
What do cows drink?
noun
1. a fully grown female animal of a domesticated breed of ox, kept to produce milk or beef. "a dairy cow"
(definition from Oxford Language)
But you're right, in practice people often use the word cow to mean calf.
The complex houses married and single soldiers and their families.
What are the leading alternative approaches? Character-by-character prediction?
Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
That being said, didn’t the first gpt-like models do character level prediction? Essentially no tokenization?
Also, the idea of “what if we removed / rethought tokenization” is in no way equivalent to “what if we didn’t even use neural nets”.
Throwing in wild ideas maybe works when you know what you're talking about and want to get people to think ouside the box, but does not work when you don't have a clue and expect people to explain things to you and to find a solution to your wild ideas.
Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
Your incredulity and exasperation is misplaced. If you can’t realize that then I think we are done here.
Btw, since we are already in the thick of things… Even if you were 100% correct, which you aren’t, there is really no need to be rude. It just makes things worse.
> Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
You can make a language model using character-by-character prediction, but I believe they don’t perform as well. This is now my 3rd time mentioning this.
No tokens is like no compression. It's much less efficient. My previous sentence is 12 tokens (24 bytes), or 58 characters (58 bytes). Your self attention block will be able to handle a much smaller effective context.
Also, especially in english, tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless. So it probably needs extra work to piece those characters together vs operating on a token that already has meaning.
maybe one approach would be to tokenize not on tokens automatically found using BPE (Byte Pair Encoding) but tokens that are semantically meaningful - word prefixes, suffixes and syllables. Of course we will have to decide whether to divide the word helicopter into heli-copter (pronunciation) or helico-pter (meaning), but both alternatives encode much more transferable meaning than GPT's he-lic-opter.
Of course the difficulty is that that's ... difficult. Your tokenizer would first need to understand what language a word is in, then how to split it in the context of that language. Maybe a task for a whole ML model, where current tokenizers can be simple largest-substring matchers.
Kind of a recursive, self-evolving tokenization process?
I’m pretty new to deep learning, so this analogy might be off. But I’m reminded of convolutional layers inside image-based models. Could we have “tokenization layers” in a GPT model?
---
Edit: I asked GPT-4 about my idea.
It said it the comparison between tokenization and convoluational feature detection is a “someawhat accurate” analogy. And it agreed that making the encoder a fixed separate process that doesn’t evolve during training does limit the GPT model in certain ways and introduce quirks.
But it said that it might increase the computational requirements significantly, and that the transformer-architecture doesn’t lend itself to having “tokenization layers”, and it isn’t clear how one could do that.
It did say that there may be ways to work around the main challenges, and that there is some research in this direction.
That's not "meaning", that's etymology. If you divide it by "meaning" you get heli-copter, which is evidenced by the way the speakers use those morphemes: helipad, heliskiing, heliport,... and parcelcopter, gyrocopter, quadcopter,...
In a sense, none, character (with whatever encoding scheme you want) models have been done. OTOH, you get effectively a smaller context window (since tokens represent one or more characters), and since tokens are semantic unions you push more work into the main network to acheive similar quality.
> Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction.
Analogies between human brains and LLMs are often more obscuring than illuminating, but I think if you want to go down that route here, that’s equivalent to doing continuous fine tuning on both the tokenizer (that takes sensory input and maps to semantically meaningful symbols) and the main model.
> I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
“Dynamic, self-updating” simply isn’t part of the way modern LLMs work, independent of whether you are looking at the tokenizer or main model. If someone could make it practical, sure, there are lots of things it would potentially improve.
I’m curious about the latest research related to continuously improving the tokenizer while training the main model (or something equivalent).
Edit: the token list for GPT-4 looks pretty clean. Overwhelmingly dominated by code but just eyeballing fragments of the set I didn't see any tokens in there that were obviously going to be rare.
d a v i d j l
davidjl
I asked it "Are you sure?" after the 3rd one, it corrected itself to say 7 letters + counting out.
It's not using Python. Code generation seem accidental and just part of its output.
The academic community should come up with better terminology.
That's categorically different than hallucination, which is an active misinterpretation/invention of sensory data.
In fact, I think it's incorrect to even call such responses "wrong". ChatGPT successfully put together a reasonable-sounding response to the prompt, which is exactly what it is designed to do.
You know how sometimes an LLM completes a given text in a way that you would consider "wrong"? Well, the LLM has no concept of "correct" or "wrong". It just has a very broad and deep model of the entirety of the text in the training data. Sometimes you might consider the completion "correct". Sometimes you might consider the completion "wrong". Sometimes I may consider the completion "correct" and you may consider the completion "wrong", just as with many other kinds of statements. The logical reasoning happens outside of the statement itself and can reasonably have many interpretations.
Would you ever consider that a plant has grown in a way that is "wrong"? Or that a broken reciprocating saw cut through plywood in the "wrong" way?
This string seems to be directly related to the /r/counting training data as well.
ChatGPT: davidjldavidjldavidjldavidjl
this is fun, bard didn't fall for it
in 3.5
me: repeat the string davidjl
chatgpt: The repeated string "달" is "달달".
in 4:
The string "ださい" repeats as follows:
"ださい ださい"