Hacker News Reader

I never understand the “only a next token predictor” argument. I do share the view that something fundamental about intelligence might be missing in LLMs, but I don’t see how this argument captures anything. If you can predict the exact next word Einstein is about to write, every time, you’re as intelligent as Einstein! It seems very clear that the LLM is also “thinking” about where the output is headed after this next token.

I do hear this position quite a lot though so I’d be interested to hear more about what I might be missing.

Next token predictor is not a very good argument. A submarine does not need to be a fish to swim.

Saying it is a 'next token predictor' should not diminish what LLMs are capable of. It is incredible what can be achieved by a simple rule like this.

Look what we built with only 2 digits: 0 and 1.

It's a statistical machine, so you only predict a statistical Einstein, ie. something that might appear quite like Einstein and even can be fairly accurate in predicting the next word he types, but which is not him, and will be unable to predict the words that are produced not just statistically but with certain intelligence, direction and intuition.

LLMs cannot, by their core design, predict next word correctly every time. That is the main thing you are missing. Another thing is that I know that I am not predicting next word based on othr words in my "trainin data", there is much deeper understanding of most subjects that I am using to "predict next word". This may not be true for everyone, I cannot know for sure, but it might be possible that there are people out there who are just biological LLMs, and in that case I am certain those people will not be able to predict next word Einstein would say either.

> If you can predict the exact next word Einstein is about to write, every time, you’re as intelligent as Einstein!

Einstein is not hold in such esteem because he was really could at spewing out text or formal proofs or theorems, but because he was able to invent something no-one else could. Some people say he advanced physics 100 years wuth his theories in relativity, whether it is true or not.

So even if you can predict what Albert is going to write next, you are still merely copying old Albert. They are amazing at processing large amounts of data, naturally, but creating something new and surprising we have yet to see evidence of (the solutions of the millennium problems seem to be brute-forcing a solution based on fuckton of data to me, but I am not a mathematician so I guess it is up in the air still).

The main criticism is, imo, that these models cannot think and have no logic. This is proven time and time again. Every day something new comes out, like counting Rs or the ”drive or walk“ test etc. These are the ”fixed“ or rather band-aided on a case by case basis, but nobody seems to be able to fox the underlying problem.

Having done my master‘s thesis on Transformer LMs, I think this is a fundamental limitation of the architecture.

These models can solve math problems and then fail to count the number of problems they solved. Not being able to count to 68 is not intelligence.

It seems you are just gatekeeping intelligence.

"Not being able to quickly reference the sum corpus of human knowledge across all fields, or contextually learn new languages in moments, or grasp the overarching architecture and modules of a million+ line codebase is not intelligence."

This would be a pretty ridiculous statement don't you think?

You are operating on a narrow, human-centric view of intelligence here for some reason... while I bet you would say an Orca is intelligent. So what drives you to place an LLM so low on the intelligence totem pole? This is clearly bias.

The fact that it is programmed is irrelevant. The fact that at it's core it is an ingestion of human data and an output of a prediction of more data doesn't declassify LLM's as intelligent. Unless you are willing to put that same lens against how humans learn. (you aren't)

Not that these distinctions are meaningful in any way, by the way, in light of raw functionalism. I am just annoyed by the clear bias.

Why would being able to quickly reference all of human knowledge be relevant at all? Intelligence is not about speed, it is about correctness, which is precisely why it's concerning that frontier models keep stumbling on things that should be piss easy to get correct.

Else by your logic the exact same model running on hardware that's half as powerful is somehow half as "intelligent"

And no, humans and other species don't learn the way that LLMs do. That's fairly obvious even if one isn't a neuroscience nerd.

“It” is not “thinking” for christ’s sake, it’s a massive statistical regression curve fit to a massive, frozen library of text. Unless you’re thinking of “thinking mode” which still is just a template that tells the model to ask questions, which are asked one token at a time, turtles all the way down.

This may be news for some, but they actually train these things with RL now, not just pretraining on a library of text.

The thinking part is still not a “natural” or “organic” process happening during inference.

> I never understand the “only a next token predictor” argument

Yeah, that's typically the case for people who have no clue about LLMs or AI.

“Predicting the next token” is a bad argument, as it completely ignores all the emergent properties of transformer models. It’s like saying that discreet electrical impulses from sodium/ potassium transporters is the beginning and end of what neuroscience describes.

Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?

I don't think the nature of 'predicting the next token' is the problem. One could argue that any organism works the same way. The problem is that we've had million years of evolution to stabilize the platform such that the important parts of it work without that many hallucinations -- and even then that sometimes fails and you get failure modes like e.g. schizophrenia, where the context window gets confusing and the 'next actions' don't make that much sense anymore, seen from the outside.

An LLM never experienced evolutionary pressure the same way. There's no fundamental difference between how it was incentivized to answer correctly when asked about how many Rs there are in strawberry vs whether it is safe to leave a house through the second story window, even though for any physical organism, those two would be _very_ differently weighted during training.

We have a world model.

> Is there any evidence that you yourself are more than a 'next action predictor with a multi-modal context window'?

I love the age of LLMs because people will post stuff like “the fact that I absolutely, unqualifiedly, fundamentally do not understand two separate things indicates that I understand at least one and maybe even both of them more than you do“ and then do that cool move where they pretend they were gonna high five you but don’t

it is a golden age of Posting

It's because of the proliferation of "insight-posting" - people thinking their little insightful "aha" thoughts are unique or have any kind of value at all.

However, insight born of knowledge ignorance is worthless. I really don't care for your great insight about nuclear fusion if you're not a nuclear engineer or physicist working on nuclear fusion, because your insight is just a hypothesis in the void that is your ignorance.

Ironically TFA falls victim to the same thing. It doesn't seem like the author is an economist or an expert of corporate finance. Their assessment of whether something is a bubble or not has absolutely no value, and carries with it the arrogant implication that the hundreds of thousands of people who are actually experts at these things never had the little 2AM epiphany that they had.

It's a smaller version of the same delusion that produces crackpots.

Is your post any better? You attack GP's character without addressing any of his arguments. You could have provided evidence for why you think his speculation is wrong, or tried to point out how his logic falls apart.

> You attack GP's character

I said the move was cool ???

I'm not even sure which part of my post you object to, or why

> entire premise on whether it is a bubble depends on if LLMs can be more than a next token prediction machine or not

I don't think current valuations need AGI to work. They just need enterprise and personal spending on AI to keep rising exponentially for a while.

We could absolutely wind up in a case where these companies are fairly close to a trillion dollars each, but not multiple trillions, at a reasonable multiple, all without AGI.

> The entire premise on whether it is a bubble depends on if LLMs can be more than a next token prediction machine or not

A theoretically perfect next token prediction machine has limitless utility. Predict the weather, stock market, lottery number, the next step of groundbreaking cancer research, etc etc. That's not the argument against current LLMs being capable of being AGI.

This is not how LLMs work.

I think you misunderstood the point. The point is that "next token prediction is not enough" is not true; in fact, it's plenty enough.

The best next chess move is ___

The best stock to buy now is ___

If that is all you have then the next chess move will be random (or whatever token is more likely in isolation), or the stock prediction will be the same.

For the chess example to work you need the current board state in the context or the move history. How an LLM performs will depend on what it has learned across the transformer layers and in the residual stream, as well as how much chess data it has been trained on. -- NN models like Leela and Stockfish have been trained on thousands/millions of games and have a game-specific state space representation.

The stock prediction example is similar, but doesn't have a consistent state space in which to make that prediction. It will need information in its context (current news articles, stock history of all stocks, etc.) in order to be able to make a prediction. Its performance will depend on whether it has been able to learn factors that make it possible to make accurate predictions.

You are completely missing the point.

The point is that next token prediction is potentially very powerful. Nobody's claiming that LLMs can predict the market or win a chess match.

But the argument against next token prediction as a powerful tool is void.

It's the transformer architecture (token embeddings and cross-token encoding) and residual stream (cross-transformer encoding and token differentiation) that make LLMs powerful tools.

The key question is not whether this architecture is powerful. The key question is how well these models can represent things outside of their design.

With chess for example, are they actually learning the rules and strategies of the game (like AlphaZero/AlphaGo, Leela, etc.) or are they just relying on a large corpus of chess games? For example, a lot of early game moves will have been played out and the best moves will likely be the more frequent moves in the training set. So LLMs could do well early game based on that game corpus.

I think that you fundamentally understand "predict" differently from those making the argument.

> The stock prediction example is similar, but doesn't have a consistent state space in which to make that prediction. It will need information in its context (current news articles, stock history of all stocks, etc.) in order to be able to make a prediction. Its performance will depend on whether it has been able to learn factors that make it possible to make accurate predictions.

This also presumes that accurate predictions are possible.

A theoretically perfect next token prediction machine is just an LLM with a temperature of 0, which performs worse than one with a non-zero temperature (i.e. imperfection)

The "prediction" has nothing to do with predicting the future, it just means choosing the most likely next token for existing human text

A theoretically perfect next token prediction machine would be uncomputable.

I wonder: Are you (or any human) any more than a next token predictor? Can you prove that you are any more than the input of your genes/environment, with a deterministic output?

Predicting the next token well is indistinguishable from intelligence. The bubble is financial, as you point out, just like railroads, radio and dotcom bubbles were. The technology didn’t die, only the companies did. LLMs are in the exploit phase, that doesn’t mean there isn’t more to explore and there may be an architecture-hardware combination that takes us an order of magnitude or two higher in a step-function-like fashion just like LLMs did 2024-2026.

The entire premise on whether computing is a bubble depends on if computers can be more than a bit-flipping machine or not. Adding operating systems, compilers, distributed clusters, and so on to enhance the results from what is essentially just flipping bits still means this technology at its core only flips bits. It is not a general-purpose machine, and unless something fundamental changes, it never will be. There are certainly valuable applications for this technology, as we have seen in ballistic tables, census tabulation, and other sectors. But it is still not more than taking in voltages and toggling transistors between 0 and 1.

> Adding reasoning agents, agent swarms, and so on to enhance the results from what is essentially just predicting the next token still means this technology at its core only predicts the next token. It is not AGI and unless something fundamental changes, it never will be.

A lot of people have dismissed LLMs four years ago for this reason. "It cannot possibly do coding since it doesn't think, it just predicts the next token." On the other hand, Scott Alexander did say in 2019 that next token prediction is a brilliant idea and might cover a lot, including math proofs [1]. So I wouldn't say it is impossible for either 1) LLMs to be all the way to AGI or 2) LLMs to become creative enough to tell us what the AGI architecture is.

[1] https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...

Software will never take off until we can show that it’s not just a bunch of simple instructions like load, store, add, and multiply.

But despite getting better and better at not making stupid mistakes, the models are still just next-token predictors. There’s plenty more room for them to improve without a totally different architecture.

> Just look at what they have done to computer science already.

I do. And it’s not good.