Comment by Fordec
21 hours ago
This sort of stuff has me believing that decision model controlled/prioritized context is the future of context management beyond the raw AGENTS.md file.
Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.
I too feel like context management has a lot of room for improvement, but the problem right now is that dynamic context management will most likely tank your cache hit rate. We need caching that's not just based on prefix-matching for this to make sense.
My bandaid fix for this is having a large number of sessions or "agents" and being ok with them being inactive at times, but they maintain their context. This has to be in some kind of system with appropriately sized "departments" to justify that agent's existence, or to justify it splitting into two, or creating sub-agents, etc. Yes, there is a cost when you cold start an agent after a week, but for now I'd rather pull that whole 400k+ context back without layers and layers of /compact. And then hopefully you keep that agent going for long enough to benefit from any cache-hit discounts.
Not really a good analogy, but it feels like having an image you like but every time you view it, it runs it through another loop of jpg compression. Use it when it's most effective, otherwise let it preserve it's details.
And as I see it that is impossible with current autoregressive global attention models because each part depends on the one before. Maybe if we can cache the local attention in blocks and only recompute the global parts.
Yes × 1000... Jev is the classic type of scientific discovery, any human can see it's effectiveness without actually trying it. We were only deprived the discovery because of the incredible noises created by the AGI/ASI ferver....
most people only need if statements
wait till they found out about that nanosecond latency “conditional model”
turns out I’ve been an “RLIF” training expert my whole career
Amusingly the biggest revolution of this tech wave will probably be the automatic if writing.