Comment by joshribakoff
1 month ago
That’s why you’ve gotta test your tests. Insert bugs and ensure they fail.
As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too.
It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!
Mutation testing is becoming the only way to catch AI red-handed. Without mutations you'll be staring at a perfect CI/CD dashboard, unaware that your tests verify absolutely nothing
Yeah, it burns CPU like crazy, but CPU time is dirt cheap right now compared to the cost of an engineer debugging that self-deception in production
but who will test the tests of tests?
The double entry technique is the most effective path to ensure accuracy (best tradeoffs for time vs accuracy) in finance and software. ie Triple book accounting has not been the standard because it's a bad tradeoff. It requires a large increase in time and effort, for rare increases in accuracy.
The mutation testing engine.
Just add tests to test your test tests