Hacker News Reader

Comment by PopePompus | Hacker News Reader

> It's a word prediction algorithm. Literally any collection of words (sometimes also known as a fact) it was trained on, regardless of how esoteric or domain specific, should generally be able to be regurgitated and, to a lesser degree, associated with similar collections of words.

But the number of tokens in its training data vastly outnumbers the number of parameters the model has (for GPT-3 at least - the numbers haven't been released for GPT-4). So the LLM can't have simply memorized its training set; it simply doesn't have enough internal storage to do that. So it is not just regurgitating portions of its training set.

View on HN

It's not about directly regurgitating the training set, but about creating associations that allow it to indirectly regurgitate, and more critically - mix, its training set. It's easier to understand this in a different domain. Imagine we want to make a letter parsing algorithm.

The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.

Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. Now change those 10,000 input pixels to words instead of pixels, with the target output being a word instead of a letter, and you get ChatGPT. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.

You just explained how abstraction and intelligence in general works.

> but can consistently replicate not only its training but even more beyond that

We call that generalization. No?

Intelligence isn't especially well defined, so you end up lost in a semantic jungle there if you don't really define what you're talking about. In this context, I think most of us are implicitly talking about intelligence such as we would associate with humans. And what is that? 50,000 years ago we knew basically nothing, and even language itself may not have been developed. Now? We've discovered the secrets of the atom, put a man on the moon, and more. We seem to have clearly gone well beyond any reasonable interpretation of what our "training set" at the time was, and at an unbelievably rapid pace if one considers how extremely inefficient our march forward (and sometimes backward) is.

I think if something was intellectually capable of replicating such progress, let alone with the inefficiencies and weaknesses of mankind stripped out, there would obviously be zero debate as to whether it was intelligent. Just think about the implications. Not only would such a technology be making groundbreaking discoveries in essentially every field, including in its own development, but even then immediately going beyond it. Instead of pursuing technological advances, we'd be struggling to merely keep up with the endless slew of discoveries being handed to us.

I agree word play is always an issue. Keep in mind we start over every generation. We keep training on new data.

I've also thought about human development and what I find interesting is that an anatomical modern human is not automatically capable of forming modern civilization (see isolated tribes still living and general hunter gatherer history in general). There is so, so much that depends on emergent factors like some sort of global or societal intelligence. I'm thinking AI has the same characteristics. It's not about the individuals, it's about "society".

> 50,000 years ago we knew basically nothing, and even language itself may not have been developed

I think language is substantially older than that, but it is a hot area of debate.

The number of tokens in the training set is not a good measure of the information content of the training set - if it was then the training would not work as the training set would be statistically random.

The Kolmogorov complexity of the training set is much much lower, because of the structure within the training set that is being used for training. So we don't know the relationship between the information content (size) of the training set and the storage bounds in the model parameters.

Reply on news.ycombinator.com