Comment by dotancohen
1 day ago
After I stopped letting Claude run tests, my token usage went from "gone by Wednesday" to "what can I automate this Saturday so my tokens won't go to waste".
1 day ago
After I stopped letting Claude run tests, my token usage went from "gone by Wednesday" to "what can I automate this Saturday so my tokens won't go to waste".
Thanks this is very helpful, i didn't realize that. But not sure what to do since running tests is how Claude often catches mistakes and iterates. If i run them myself, the cache will still expire.
However where i think you have a point is that it tends to run tests quite casually where a reviewer would have been enough to catch mistakes.
IMHO, the real problem is the 5min cache way too small. They should up it to 1 hour.
With a subscription the main conversation has a 1h TTL, and subagents 5m. https://code.claude.com/docs/en/prompt-caching#cache-lifetim...
Oh makes sense thanks, so i'll forbid agents from running builds and tests.
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.
I don't actually know the real mechanism, I came across this serendipitously.
By default Anthropic's token cache TTL is 5 minutes. If your tests take >5m to execute, when the LLM resumes your entire context is a cache miss
Will try to remember this next time I'm told that devs don't care about performance.
There are many reasons why your test suite shouldn't take >5m to execute. This is just one of them.
2 replies →
Do you have a link to the docs for that? Thanks.
How do you do that?
I just told it that I want to run the tests, because so much of the code was vibed at that point and I felt I was losing understanding of the application. A few days of that and I could see that my token use was much lower than usual. Going back to LLM-run testing returned the high token usage.