That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Yes, it adds tests and makes them pass. But it adds tests for things that no human would ever add. I regularly wind up removing tests where it is guarding dumb crap like "facts" in markdown files, or "drift" across files. Today I came across a file where it was walking ASTs across multiple languages to ensure that there was no drift. I get why it'd do that, but simple changes become absurd.
I do. My experience has been that most people don't. And then the ridiculous tests become blockers for the agent as they believe the tests are enforcing "truth", which may not be correct post-evolution.
Anecdata, but most of the time when something doesn't smell right to me looking at genAI code, the root cause is ridiculous tests that made their way into the repo. And then the agents built up around it.
That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
The only time I really get tests which take crazy time is when I'm doing mobile dev, because emulator testing on my system is slow.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
And I’m definitely not saying don’t do that.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
And then you discover it’s added tests for insane things that shouldn’t be getting tests
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Yes, it adds tests and makes them pass. But it adds tests for things that no human would ever add. I regularly wind up removing tests where it is guarding dumb crap like "facts" in markdown files, or "drift" across files. Today I came across a file where it was walking ASTs across multiple languages to ensure that there was no drift. I get why it'd do that, but simple changes become absurd.
3 replies →
Then you remove them and move on with your life? We're still software developers at the end of the day, my delete key works better than ever.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
I do. My experience has been that most people don't. And then the ridiculous tests become blockers for the agent as they believe the tests are enforcing "truth", which may not be correct post-evolution.
Anecdata, but most of the time when something doesn't smell right to me looking at genAI code, the root cause is ridiculous tests that made their way into the repo. And then the agents built up around it.