Hacker News Reader

Comment by mlmonkey | Hacker News Reader

Anecdotally (sample size == 1), I have observed that as my context grows longer, the model seems to get dumber. I'm not an advanced user with harnesses, etc.; I just use the chat interface, as that's all I can afford, but after a few turns the model (I'm talking frontier models here from OAI and Ant) seems to just do stupider things.

View on HN

DeepSeek v4.1 flash seems to do this thing where it starts saying that we should wrap up the current work and start again "next session" (whatever that means), that the "context budget" is almost up, or some other wording I'm forgetting, as the context gets longer.

I've taken to just manually making it write a handoff file and clearing context completely. I'm pretty much manually compacting instead of my harness doing it. Feels weird, but it does work.

I think it’s trained into them so they emit a stop token with enough window left to generate a compaction summary

Any idea what the ~rule is or how it's meant to work? I assume it'd work better if I can wire up my harness to handle it, but no idea. It didn't seem to know itself, from it's training. I didn't search too much yet though.

You could also increase compaction.reserveTokens in ~/.pi/agent/settings.json to make pi compact more often. Also you can use /compact <instructions> to specify what facts should be included in the summary.

One hard part is that from a handoff file you only have information for 1 session, so you have to manually determine next week which facts are still true. At Substrate we solve this problem using our memory: we have 1 memory outside of an agent and track the source of every fact. https://trysubstrate.co/l/a2hn

after thinking about it more, in the attention there is some positional encoding, so I think it's just that tokens are attending to this as well.

I would assume it's the harness sending the model a notification toward the end. At least Claude Code is designed to attach reminders with tool call output as a way for the harness itself to steer the model slightly.

Relatively sure it's not. I'm using pi and I can't find any place it's doing any such thing. My current theory is that it's behavior trained into the model either accidentally or intentionally, probably during post-training in some harness that _does_ give hints of that sort?

Let's bump that sample size to 2. I've observed the same and that observation has heavily influenced the design of my coding agent harness.

I have auto-compact on at 400k

I try to manually manage as much of the context as possible. Most of the time, I limit myself to a context window of 200k, and it works great for my use cases.

I can't cite the sources at the moment but this is already well known - a few papers on it and Anthropic say somewhere in their docs that you want to get across the minimum amount of context required to achieve your goal. Quality diminishes the more you dump in.

I've also found, as the model context gets larger, my work to supervise increases.

Like a see-saw - we just need that balance, man!!!

That calls lost-in-middle issue with LLM models. You have to use appropriate workflow patterns like map/reduce to handle big context. But be sure to run experiments on dataset first to understand what logic works well and stable. Where NOT! And how it can be combined to reduce LLM stohastic problems in prod

It does, and, when combined with Claude's stingy cache TTL of 5 minutes (unless you up it to 1h at a sizeable increase in token cost) for API billing, means that some sessions cost like 10x what they should because of cold caches.

Reply on news.ycombinator.com