Hacker News Reader

Comment by bunderbunder | Hacker News Reader

Humans get an understanding of large scale systems into their head by developing and refining abstract models, and committing them to memory.

LLMs don’t do that. Any abstraction they do happens at training time; they cannot dynamically construct new models at inference time. Nor can they commit them to long term memory, because LLMs don’t have such a mechanism. Which, in turn, precludes progressive refinement.

We get around this by using agentic harnesses to pack everything into the context window. (Which itself is something humans don’t really have.) This is why large context windows are so important for most real world tasks: it’s the only way LLMs have to handle information that wasn’t in their training corpus.

View on HN

I am not sure you're correct. Yes, from the physical perspective of LLMs design, the above is correct. But they are also systems that evolved to do things through training; so you cannot discount the possibility they can emulate building of new abstractions in another level.

LLMs already have the concept of "reference" - i.e. I can say a name of something, for example of a function, and they understand that the properties associated with the name itself are not relevant, but rather whatever the name does refer to.

But this is pretty much what you need to build new abstractions. You need to name a paragraph with the definition (explanation) of a concept, and be able to recall the paragraph by the name in order to do inference.

So with references, they can get around the context size limitations, like humans can. It's slower than learning things "by heart", but I think LLMs are perfectly capable of that; and the evidence is, they are able to navigate codebases larger than their context, which often contain many new abstractions.

I can say what I was getting at quite definitively. An LLM is, to an approximation, a pure function. No side effects, no state changes. No neuroplasticity. No working memory. It really is just predicting next tokens based on previous tokens. So the only way to change its output for a given input is to change the input - that is, pack different information into the context window.

Some nondeterminism from the probabilistic token sampling and distributed inference, of course. But sometimes randomly choosing the second most likely next token isn’t going to manifest a cognitive feedback loop.

Yes, but we are really talking about LRMs not LLMs. They can emulate working memory so well that many people even recommend using bunch of .md files (as a sort of out-of-context notes) instead of RAG based on vector DBs. They can maintain context, by compressing it and writing out memories quite autonomously.

So yes, even though not updating their weights, they can learn plenty new things and concepts, and build new models of the world.

But now you’ve looped back around to my original point: a large context window is essential to making it work well. Those *.md files won’t do anything if you don’t feed them into the LLM at inference time.

(edit: That said i do agree with the idea of designing systems so that such a large window isn’t needed. But that seems a bit self defeating if the goal is massive development acceleration with agentic coding. These models seem to be pretty bad at engineering for simplicity. And protecting system design with close human oversight would mean we are still limited by the speed at which humans can read and understand code.)

Large enough context window is enough. Yes, the md files can be filled back in, but other stuff (unneeded details) can be swapped out.

Gotcha. My apologies. It sounds like I misunderstood your original point and I've been talking past you this whole time.

I've been experimenting with improving coding agent performance by improving written documentation. Anecdotally I've seen that, compared to the vibecoded mess I inherited, improving the docs let us use smaller models and get better quality work out of them.

Replacing our skill files with Markdown documentation also made a difference. My hunch is it's because skills tend to get mass dumped into the context window, whereas the agent harness can make more judicious decisions about how to include that information.

But I haven't seen much indication that any of this affects any model's tendency to make a mess of the code, regardless of its context window size. Or the documentation, for that matter. Maybe we can find a hack to get them to do that well, too. I certainly hope so, because I can't see how this context window size red queen's race is sustainable.

Reply on news.ycombinator.com