So yes, even though not updating their weights, they can learn plenty new things and concepts, and build new models of the world.
(edit: That said i do agree with the idea of designing systems so that such a large window isn’t needed. But that seems a bit self defeating if the goal is massive development acceleration with agentic coding. These models seem to be pretty bad at engineering for simplicity. And protecting system design with close human oversight would mean we are still limited by the speed at which humans can read and understand code.)
I've been experimenting with improving coding agent performance by improving written documentation. Anecdotally I've seen that, compared to the vibecoded mess I inherited, improving the docs let us use smaller models and get better quality work out of them.
Replacing our skill files with Markdown documentation also made a difference. My hunch is it's because skills tend to get mass dumped into the context window, whereas the agent harness can make more judicious decisions about how to include that information.
But I haven't seen much indication that any of this affects any model's tendency to make a mess of the code, regardless of its context window size. Or the documentation, for that matter. Maybe we can find a hack to get them to do that well, too. I certainly hope so, because I can't see how this context window size red queen's race is sustainable.