Comment by michaelt
3 hours ago
I find the problem with "lots and lots of testing" as a solution - (assuming by that you mean lots and lots of automated testing) is it invites lots and lots of lines of code - making code reviews even more challenging.
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
Might be a little controversial to say this but automated tests can be reviewed at a lower bar than production code.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
IMO it works better if you split the tests into the ones you care about, and the ones the LLM writes. I assume its tests help it write code with fewer errors, so I'm happy for it to keep doing that, but they're akin to CoT - an artifact of the process that I don't need to inspect in detail.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
1 reply →
Accumulating tech debt in the tests will come back to haunt you in the long run when tests become harder to maintain or less reliable as the project evolves.