I have a different take on it, which I've been saying for years at this point. DeepMind appear to have over-committed to the wrong approach - reinforcement learning. They've had some great successes with it for sure, but it's always seemed the wrong approach for intelligence, notwithstanding luminaries such as David Silver insisting that "Reward is Enough".

The best examples of intelligent systems we have at this point are animal brains, which per cortical architecture(s) appear to operate on an entirely different principle - prediction. The cortex simply put is a prediction machine - not only can it predict, but this is essentially the only thing it does, and ability to predict is the evolutionary selection pressure that created it.

It's a bit odd that here in 2023 there is still no widespread agreement in the scientific community how best to define intelligence, but I have my own rather pithy definition that I believe is the most reductive:

"Intelligence is the ability to correctly predict (future outcomes) based on prior experience."

In other words, intelligence is the capability given to use by our cortex.

Why evolution put such a high value on prediction should be pretty obvious : you can now act based on what is going to happen rather than what you immediately perceive. Predict where the food/water is, how the predator/prey are going to behave, etc, etc. Prediction is also the basis of "thought/reasoning" which is essentially just a sequence of (correctly predicted) chained what-if scenarios.

It should now be obvious why something as dumb/simplistic as an LLM shows such a "surprising" degree of intelligence - because it is a prediction machine, and why "think step by step" is therefore such a powerful tool.

There are probably a dozen or more alternate definitions of intelligence that have been offered by various people, but I believe none are as reductive as mine - other definitions basically describe capabilities that a system/person would have if they were able to predict.

Reinforcement learning seems to be a misleadingly attractive idea because subjectively goal-based behavior makes sense to us, but it seems that again this is not reductive enough. The appearance of goal-based behavior is something that comes out of prediction. Part of understanding this is a brilliant insight from the much maligned Jeff Hawkins - prediction (quite literally and mechanistically) becomes action; this occurs because the motor cortex operates just like the rest of the cortex and is a prediction machine - it predicts our motor actions, and those predictive outputs drive our muscles causing the prediction to become true. Goal-based behavior is therefore based on the mechanism of prediction - predict the sequence of what-if actions to achieve a goal, and those actions will be taken. Our self-predictive actions start at birth by predicting (via self-observation) the actions we make at that stage and evolves as life experience causes our predictions to change.

TL;DR - DeepMind are riding the wrong horse (RL). Prediction is all you need.