A theoretically perfect next token prediction machine has limitless utility. Predict the weather, stock market, lottery number, the next step of groundbreaking cancer research, etc etc. That's not the argument against current LLMs being capable of being AGI.
The "prediction" has nothing to do with predicting the future, it just means choosing the most likely next token for existing human text
The best next chess move is ___
The best stock to buy now is ___
For the chess example to work you need the current board state in the context or the move history. How an LLM performs will depend on what it has learned across the transformer layers and in the residual stream, as well as how much chess data it has been trained on. -- NN models like Leela and Stockfish have been trained on thousands/millions of games and have a game-specific state space representation.
The stock prediction example is similar, but doesn't have a consistent state space in which to make that prediction. It will need information in its context (current news articles, stock history of all stocks, etc.) in order to be able to make a prediction. Its performance will depend on whether it has been able to learn factors that make it possible to make accurate predictions.
This also presumes that accurate predictions are possible.
The point is that next token prediction is potentially very powerful. Nobody's claiming that LLMs can predict the market or win a chess match.
But the argument against next token prediction as a powerful tool is void.
The key question is not whether this architecture is powerful. The key question is how well these models can represent things outside of their design.
With chess for example, are they actually learning the rules and strategies of the game (like AlphaZero/AlphaGo, Leela, etc.) or are they just relying on a large corpus of chess games? For example, a lot of early game moves will have been played out and the best moves will likely be the more frequent moves in the training set. So LLMs could do well early game based on that game corpus.