Comment by mlmonkey
10 hours ago
Anecdotally (sample size == 1), I have observed that as my context grows longer, the model seems to get dumber. I'm not an advanced user with harnesses, etc.; I just use the chat interface, as that's all I can afford, but after a few turns the model (I'm talking frontier models here from OAI and Ant) seems to just do stupider things.
That calls lost-in-middle issue with LLM models. You have to use appropriate workflow patterns like map/reduce to handle big context. But be sure to run experiments on dataset first to understand what logic works well and stable. Where NOT! And how it can be combined to reduce LLM stohastic problems in prod
DeepSeek v4.1 flash seems to do this thing where it starts saying that we should wrap up the current work and start again "next session" (whatever that means), that the "context budget" is almost up, or some other wording I'm forgetting, as the context gets longer.
I've taken to just manually making it write a handoff file and clearing context completely. I'm pretty much manually compacting instead of my harness doing it. Feels weird, but it does work.
I think it’s trained into them so they emit a stop token with enough window left to generate a compaction summary
I would assume it's the harness sending the model a notification toward the end. At least Claude Code is designed to attach reminders with tool call output as a way for the harness itself to steer the model slightly.
Any idea what the ~rule is or how it's meant to work? I assume it'd work better if I can wire up my harness to handle it, but no idea. It didn't seem to know itself, from it's training. I didn't search too much yet though.
I can't cite the sources at the moment but this is already well known - a few papers on it and Anthropic say somewhere in their docs that you want to get across the minimum amount of context required to achieve your goal. Quality diminishes the more you dump in.
Let's bump that sample size to 2. I've observed the same and that observation has heavily influenced the design of my coding agent harness.
3
I have auto-compact on at 400k
4
I try to manually manage as much of the context as possible. Most of the time, I limit myself to a context window of 200k, and it works great for my use cases.
1 reply →
It does, and, when combined with Claude's stingy cache TTL of 5 minutes (unless you up it to 1h at a sizeable increase in token cost) for API billing, means that some sessions cost like 10x what they should because of cold caches.