Comment by jgalt212
21 hours ago
These "solutions" place a lot of faith in a "complete" set of test cases. I'm not saying don't do this, but I'd feel more comfortable doing this plus hand-generating a bunch of property tests. And then generating code until all pass. Even better, maybe Claude can generate some / most of the property tests by reading the standard test suite.
Well they also shadowed production traffic and fixed some bugs that were causing mismatching results. Not saying that stuff can't still slip through, but it's a good way to evaluate it against real data in a way you can't from just test cases alone
parallel execution that auto-generates test cases from exceptions is very slick. That being said, you still need humans in the loop as sometimes the oracle is not THE oracle.