Comment by tasuki
9 hours ago
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
9 hours ago
What is the reason for this? Surely running tests by itself isn't necessarily token-expensive? And it provides cheap feedback for the agent: to see whether they're breaking things.
My guess is that the LLM (usually Sonnet or Opus in my case) is burning tokens polling to see if the output finished - and there is often a lot of it. My current project is Rust, Python, and Kotlin for what it's worth.
I don't actually know the real mechanism, I came across this serendipitously.
By default Anthropic's token cache TTL is 5 minutes. If your tests take >5m to execute, when the LLM resumes your entire context is a cache miss
There are many reasons why your test suite shouldn't take >5m to execute. This is just one of them.
I've worked on projects where the test suite would take hours.
.NET solution having 120+ projects, with 7000 tests of which maybe 1500 are integration and/or component tests.
Buid alone takes 4 minutes. Test suite heavily parallelized runs in about 2 to 3 minutes, infrastructure dependencies fully containerized using podman, no external API call.
And this is without mentioning the 5000 frontend tests (angular), including Playwright end-to-end tests wired to the backend (about 3 to 4 minutes run without the e2e tests).
And this is running on a dev machine with 64gb of RAM and an NVMe disk.
Some systems are just massive, and even if you run focused tests, it's easy for a change to have a large blast at least requiring a large rebuild, and at times to rerun a larger set of tests.
Do you have a link to the docs for that? Thanks.