Comment by rustyhancock
1 day ago
Regression tests?
I think people are assuming the worst but AIs most helpful contribution beyond churning out code is explaining the code base.
Who knows if they will do a good or bad job that's down to the engineers and not the tools imo.
At my previous place, we had to write unit tests to cover each and every line of code. So if you add 10 lines of code, you ended up writing 100 lines of unit tests, mocking this service and that service, etc.
Once AI came along, I became like Captain Picard (RIP) and would just sit back and let the AI write all unit tests. "Try harder, make sure coverage is 100%". It was very exhilarating to sit back and watch the models churn away, creating 100s of lines of unit tests. Of course, I checked them all manually before submission ;-)
I think this is actually the opposite in practice. LLMs generate better tests than they are at assessing which tests are effective (and, inferentially, how a system should behave). See: https://arxiv.org/abs/2410.21136
That said, there's also research pointing to LLMs encoding existing bugs as test requirements: https://arxiv.org/abs/2607.05139
That CS PHD researcher, Konstantinou, in particular, has written a lot on this: https://arxiv.org/search/cs?searchtype=author&query=Konstant...
Why is that better than having the agent(s) write regression tests or explanations of the old codebase?