Comment by kristjansson
9 hours ago
And the tests. Oh god the tests. Personal recent favorite: I asked for some changes to a Dockerfile, which it did ably, and then promptly tested by writing a pytest module that traversed up to the root, read the Dockerfile, and checked that the added lines were present.
Try property based testing perhaps.
Code-generating robots are pretty bad at property-based testing, in my experience. They can do it but they still need a lot of hand-holding. They often regress to writing a mirror implementation as the oracle and trying to enumerate a fixed set of examples they find meaningful.
I'm generally happy with the tests I ask it for, some of which are PBT. It's just the insistence upon memorializing every single change with a test. Maybe encouraging / forcing PBT will dissuade it?
I’ve heard of this, but I’m not really sure at all how to even get started. Are there any good guides out there?
Many people like https://fsharpforfunandprofit.com/series/property-based-test...
In this day and age, https://hypothesis.works/articles/claude-code-plugin/ might be useful.
I regret this but at some point I stopped reading generated tests. It feels pointless when our test files are already tens of thousands of lines of — at best — tautological slip which says that the codes does what it does.