Comment by andai
6 hours ago
I keep posting this but it keeps being relevant. I had an agent implement a feature completely backwards. It wrote a whole bunch of tests proving the correctness of the implementation. All the tests passed.
The really interesting thing to me is that formal verification wouldn't have helped there either -- it would have just written a mathematical proof of the correctness of the backwards feature.
A few models ago it was not unusual to find that Claude had written tests whose assertions looked correct, but were so thoroughly mocked out that they ran no real code at all — a Potemkin test.
I haven’t seen it do that in quite a while, but it was an interesting failure mode!
What do you mean by "completely backwards"?
In test-driven development, you're supposed to write out the tests first to specify the desired behavior, then write the code to confirm to that.
I think he means it was very wrong.
Wait, did you let the same agent write the tests that wrote the implementation? Or was it that the plan was the wrong way round, and both the generator of code and the generator of the tests followed that same incorrect plan?
Also, what were your mechanisms for reviewing the plan against the described business outcome? Anything else you could have done there to catch the backwardness?
And finally, did you run any of it across models from different foundation labs? I'll often run important stuff generated by codex across Anthropic, grok, and Gemini with an opus or fable judge...