I am interested in if it's a fundamental limitation, ie models that have gaps that need to be corrected are going to be bad at recognizing those gaps, and leaving instructions to fix them. Or if models just aren't trained on this
It seems akin to my experience with coding agents, where models are good at implementation but not great at abstraction
Firstly LLMs need to have more training on speaking to several different people.
Humans have a lot of training on interacting with many different individuals and LLMs feel extremely one thread chat heavy. Like if you read the entire thread the LLM langauge is usually consistent but if you send someone a partial artifact it feels disconnected in a broken way.
This includes their code changes and doc changes which inheret this lack of audience.
For the more general idea of keeping good artifacts for themselves I think an interesting idea would be to look into context language models instead of autoregressive ones.
https://arxiv.org/abs/2609.37725
Context language models need to manage their own context to complete a task. (They see their own context as a file).
This naturally forces this innate ability of managing relevance.
The article explicitly excludes this from consideration early on though:
By “agent memory,” we mean knowledge that Claude gains from experience...saved as context it can read later. It excludes other ways models can learn or manage context, such as changes to their weights
Those seem fundamentally different processes, and adjusting weights does seem a much better idea after all, but then in that world everyone has a different AI and how do you apply updates? For a company like Anthropic this is of course a big problem and that is probably why they don't like that route. Could we store weight deltas or something similar and reapply them over a different model?