Comment by bunderbunder
3 days ago
I tried that this summer. I encountered this doom loop where the review agent would find bugs, the coding agent would fix them in some overengineered way that introduced new bugs, the review agent found them, the coding agent made more while fixing them, repeat until insanity.
Not every time. But often enough.
FWIW gpt-6.1-sol and opus 5.5 are much, much less prone to this, if you include some coaching around complexity, YAGNI, that kind of thing
I've been building a side project end-to-end with agents, and it's really not true for complex software.
At a micro scale their code is right, but tons of invented complexity no sane person would introduce. I've found the two most common triggers are the models polluting their own context windows (poorly worded instructions in comments/issues/AGENTS.md) and a poor "taste" for breaking changes (they'll constantly try to make their additions non-breaking, and if prompted they can make breaking changes, it has to be very carefully scoped or they start assigning great reward to ripping out things)
Same experience. It's especially bad with Claude and anything with non-code text, where it will happily write up endless Claude-ese comments that break any rules in the repo and are constantly incorrect in small ways, catch them in the review, rewrite specific lines in ways that are still wrong despite still having the review in its context, and repeat twenty times in a row.
Yeah, you got to reinforce through prompting (which isn't dead!) at multi touch-points to make narrow bug fixes and not touch unrelated code.
And as others have said some models people prefer for implementation work are less prone to this.. 5.6 Sol, 6.1 Sol, Opus 5.5
Agents understand red/green/refactor. Tell them to follow it, and use a coverage tool.
That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
The only time I really get tests which take crazy time is when I'm doing mobile dev, because emulator testing on my system is slow.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
1 reply →
And then you discover it’s added tests for insane things that shouldn’t be getting tests
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
4 replies →
Then you remove them and move on with your life? We're still software developers at the end of the day, my delete key works better than ever.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
1 reply →