Comment by bryanlarsen
7 days ago
I used this to great success just this morning. I told the AI to write me some unit tests. It flailed and failed badly at that task. But how it failed was instructive, and uncovered a bug in the code I wanted to test.
In a way, AI’s failure can be its own kind of debugger. By watching where it stumbles, you sometimes spot flaws you’d have missed otherwise.
Haha, that's awesome! Are you going to change the interface? What was the bug?
It used nonsensical parameters to the API in way that I didn't realize was possible (though obvious in hindsight). The AI got confused; it didn't think the parameters were nonsensical. It also didn't quite use them in the way that triggered the error. However it was close enough for me to realize that "hey, I never though of that possibility". I needed to fix the function to return a proper error response for the nonsense.
It also taught me to be more careful about checkpointing my work in git before letting an agent go wild on my codebase. It left a mess trying to fix its problems.
Yeah, that's a perfect example of what I'm talking about!