Comment by ozten
14 days ago
I got massive productivity gains from having an LLM fill out my test suite.
It is like autocomplete and macros... "Based on these two unit tests, fill out the suite considering b, c, and d. Add any critical corner case tests I have missed or suggest them if they don't fit well."
It is on the human to look at the generated test to ensure a) they are comprehensive and b) useful and c) communicate clearly
Can you extend that - what was the domain, how did you start? I would like to give this a try but am not quite sure I get it?
Backend coding for web services.
In the past I would hand write 8 or 9 unit tests. Now I write the first one or two and then brain dump anything else into the LLM prompt. It then outputs mine plus 6 or more.
I delete any that seem low value or ridiculous or have a follow up prompt to ask for refinements. Then just copy/pasta back into the codebase out of the chat.
That simple ? I’ll try it
Can confirm this approach works well for us too.
See, I’m arguing for writing fewer, better tests.
I realize that it’s the norm to rely heavily on unit tests. Hundreds or thousands of examples of inputs and outputs. We still find errors in programs. “Examples prove the presence of an error, not the absence of errors,” as Djikstra (or was it Hoare? I can’t remember) would say. So I understand how one could view having an LLM generate tests being a win for productivity in that case.
But such test suites don’t add much. And generating 20 more tests won’t tell me much more about the code. It will actually make the test suite harder to read and understand.