Comment by michaelt
8 hours ago
I find the problem with "lots and lots of testing" as a solution - (assuming by that you mean lots and lots of automated testing) is it invites lots and lots of lines of code - making code reviews even more challenging.
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
I take the complete opposite approach which is to have zero unit tests now, because LLMs are excellent at both reviewing code to understand what's going on, and also doing manual QA given environment instructions. Lots of unit tests will just bloat the context which ultimately dilutes the quality of the LLM's work and also creates additional work (it has to update the code and the tests each time).
Unit testing imo is pre-LLM. It was mostly to keep the human honest because the human can't possibly read all the code to make sure nothing broke.
> LLMs are excellent at both reviewing code to understand what's going on ...
Have you ever worked on existing code base that has components that makes no sense but must be there?
I'm flabbergasted at this comment.
Are you maintaining a list of all invariants you want QA'd by the LLM after each change? How many tokens does this process take?
You want tests to be obvious "at a glance". And then some of the tests should at least indicate it's covering (for the most coverage of the system it reasonably can) the cases we care about.
The name of the test should indicate what it's doing.
For unit tests, these should check only one thing, and the test can be structured into "assemble, act, assert" style tests.
On the implementation side: code which touches state is tricky. -- So, minimize the amount of code which touches state (or rather, shift as much code as you can to stateless code, which is easier to bring under test).
Might be a little controversial to say this but automated tests can be reviewed at a lower bar than production code.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
IMO it works better if you split the tests into the ones you care about, and the ones the LLM writes. I assume its tests help it write code with fewer errors, so I'm happy for it to keep doing that, but they're akin to CoT - an artifact of the process that I don't need to inspect in detail.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
1 reply →
Accumulating tech debt in the tests will come back to haunt you in the long run when tests become harder to maintain or less reliable as the project evolves.