What can a transformer not do that people say they can do?
The parent comment said, paraphrasing, "learn from interaction with the world", and I'm responding, they absolutely can already do this by taking their logs of interaction with humans and updating their weights through backprop.
The reason you don't see that done "live" is primarily an economics problem rather than a limitation of the model structure.
Because backprop fundamentally wants the entire data set in every pass. It doesn't behave well and is destructive when you update after the pre-training phase. RLHF/LORA are attempts to work around that and effective at what they do, but it is not learning in the sense you are talking about and also do not fully address the catastrophic forgetting problem. This architecture as is - is not compatible with continual learning.
It IS done with every new model release. Why do you think SpaceXAI bought Cursor? And then immediately had a huge jump in capability with their next model version?
It's just done in large batches for economic reasons.
> backprop fundamentally wants the entire data set in every pass
I'm genuinely unsure what you mean, it's not even possible to run backprop in this way?
Bolting on new data to an existing model (fine-tuning) is precisely what gets you catastrophic forgetting.