Comment by William_BB
2 days ago
> AI is also scarily good at writing tests
:-) I hope you read those tests before claiming it's "scary good"
2 days ago
> AI is also scarily good at writing tests
:-) I hope you read those tests before claiming it's "scary good"
Indeed, much of the scariness is how fearlessly and confidently it writes them with little regard to their actual usefulness or value. When I find it adding a lot of tests, I often say something like: "audit each test carefully, and consider whether the test is testing a meaningful boundary or is more ceremonial. delete low-value tests and add new tests to cover meaningful boundaries not exercised by the gaps you identify". Without fail, this always produces some decent results.
Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
I notice that the tests are very often for its own benefit. Like, you’ll ask it to stop doing something, literally to just remove code, and it will write a test to verify the behavior is not there.
I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
Lots of unit test do not add value in the traditional sense, but they do help the llm to understand the code.