Transformers are sample-efficient world models

You wonder if that means intelligence really works like other bodily systems. Animals work so well not because of a single trick, but due to a big bag of tricks. Not "one single gene" or algorithms learning good performance at everything, but 1000 different algorithms and a good way of selecting between them. Not 1 or 2 or 10, but a large number.

Reinforcement learning has an issue currently with modeling complicated aspects of an agent’s given simulation (their “world model”) over long time horizons.

This is probably because there is a vast amount of information in any given “frame” of this simulation and the agent has no baked-in priors about its environment (or if they do, it’s because they’ve been hand-programmed, which isn’t ideal in a field premised on the automation of such things).

If you instead pretrain _another_ model using the same tech as GPT3 on the agent’s environment, the agents can use this model during their simulations to model the world they see, essentially giving them something that looks kind of like common sense (although I certainly wouldn’t call it that to another researcher).

Ok, let's have one.