Comment by kaelwd
11 hours ago
https://github.com/k0valik/pi-blackhole works pretty well for this, instead of a summary of a summary of a summary it gets a rolling log of important instructions and discoveries, and can /recall exact contents of a previous message or tool call if needed.
That looks interesting, have you found it useful in practice? I'm a bit burnt out on these wildly ambitious big plugins (headspace, RTK etc.) because in each case when I've done personal evals they come out worse than the default.
I haven't done a proper benchmark or anything but it isn't noticeably worse at least, and the instant compaction with background processing is nice for slow models.
> it isn't noticeably worse at least
That's what I thought about RTK and Headroom too but when I did some fairly basic evals across a small set of tasks (a matrix of RTK, Headroom, both, or off) it came back very clear that off was best. I didn't dig into why too deeply due to lack of time, but it looked like the token reduction per turn was being replaced by the model needing more turns.
That and building my own pi extensions made me realize there's a lot of subtly here. I can build a tool that on paper works perfectly but tuning the system prompt to get it used correctly takes time and evals.
The problem is, nobody is doing them, or not well, and not reproducibly.
This looks especially hard for something designed for very long running tasks. We'd need to set up one version using normal compaction, one using this, across several tasks, leave them running for hours (burning tons of tokens) and then repeat at least a couple of times. If we don't do that, we're just guessing.