Comment by esperent

> it isn't noticeably worse at least

That's what I thought about RTK and Headroom too but when I did some fairly basic evals across a small set of tasks (a matrix of RTK, Headroom, both, or off) it came back very clear that off was best. I didn't dig into why too deeply due to lack of time, but it looked like the token reduction per turn was being replaced by the model needing more turns.

That and building my own pi extensions made me realize there's a lot of subtly here. I can build a tool that on paper works perfectly but tuning the system prompt to get it used correctly takes time and evals.

The problem is, nobody is doing them, or not well, and not reproducibly.

This looks especially hard for something designed for very long running tasks. We'd need to set up one version using normal compaction, one using this, across several tasks, leave them running for hours (burning tons of tokens) and then repeat at least a couple of times. If we don't do that, we're just guessing.