Comment by satnhak
4 days ago
I once worked in a company where there was an 80% code coverage requirement. Some enterprising contractor had a script that generated a single file with its own covering test suite the size of which could be tuned to achieve 80% over the whole codebase. Mostly the code was untested.
And thanks to AI, we could generate extremely convincing reams of code whose only purpose is to be fake unit tested. Amazing. I sincerely hope I never need to use this nuclear weapon.
Or better yet: effectively fake unit tests. It is almost never the case that tests written by AI detect actual issues. At most they detect that has changed.
Yeah I’ve been thinking that LLM unit tests are basically snapshot tests. Just sorta ossify things in place. If they break, you just ¯\_(ツ)_/¯ and have the LLM fix them. It’s like they were never there!
So it’s just like the olden days of everybody ignoring tests, but we give anthropic a ton of cash