Comment by kristjansson
10 hours ago
And the tests. Oh god the tests. Personal recent favorite: I asked for some changes to a Dockerfile, which it did ably, and then promptly tested by writing a pytest module that traversed up to the root, read the Dockerfile, and checked that the added lines were present.
Try property based testing perhaps.
Code-generating robots are pretty bad at property-based testing, in my experience. They can do it but they still need a lot of hand-holding. They often regress to writing a mirror implementation as the oracle and trying to enumerate a fixed set of examples they find meaningful.
I'm generally happy with the tests I ask it for, some of which are PBT. It's just the insistence upon memorializing every single change with a test. Maybe encouraging / forcing PBT will dissuade it?
I’ve heard of this, but I’m not really sure at all how to even get started. Are there any good guides out there?
Many people like https://fsharpforfunandprofit.com/series/property-based-test...
In this day and age, https://hypothesis.works/articles/claude-code-plugin/ might be useful.
I regret this but at some point I stopped reading generated tests. It feels pointless when our test files are already tens of thousands of lines of — at best — tautological slip which says that the codes does what it does.
I think reading and writing test code is harder than the underlying code itself. You must know both the desired behavior of the code-under-test and whether the test correctly stresses that behavior. And then if you're working on anything more complicated than a single unit test you must also make sure that it doesn't blow up anything adjacent to it (preserve state).
I think enforcing black box testing is the best way to get useful tests out of both humans and robots. They must not know the internals, or it will lead them to do bad things.