That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Yes, it adds tests and makes them pass. But it adds tests for things that no human would ever add. I regularly wind up removing tests where it is guarding dumb crap like "facts" in markdown files, or "drift" across files. Today I came across a file where it was walking ASTs across multiple languages to ensure that there was no drift. I get why it'd do that, but simple changes become absurd.
Sounds like better guidance needed. You have to always keep in mind that the agent doesn't understand the repo from the "business" perspective, only from the functional. You have to give it the why, and ensure a shared understanding. Whenever I plan epics with an agent I always check on acceptance criteria before the tickets hit the tracker, so I know it gets the business aspect. Tends to lead to far better code.
I do. My experience has been that most people don't. And then the ridiculous tests become blockers for the agent as they believe the tests are enforcing "truth", which may not be correct post-evolution.
Anecdata, but most of the time when something doesn't smell right to me looking at genAI code, the root cause is ridiculous tests that made their way into the repo. And then the agents built up around it.
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Yes, it adds tests and makes them pass. But it adds tests for things that no human would ever add. I regularly wind up removing tests where it is guarding dumb crap like "facts" in markdown files, or "drift" across files. Today I came across a file where it was walking ASTs across multiple languages to ensure that there was no drift. I get why it'd do that, but simple changes become absurd.
Sounds like better guidance needed. You have to always keep in mind that the agent doesn't understand the repo from the "business" perspective, only from the functional. You have to give it the why, and ensure a shared understanding. Whenever I plan epics with an agent I always check on acceptance criteria before the tickets hit the tracker, so I know it gets the business aspect. Tends to lead to far better code.
2 replies →
Then you remove them and move on with your life? We're still software developers at the end of the day, my delete key works better than ever.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
I do. My experience has been that most people don't. And then the ridiculous tests become blockers for the agent as they believe the tests are enforcing "truth", which may not be correct post-evolution.
Anecdata, but most of the time when something doesn't smell right to me looking at genAI code, the root cause is ridiculous tests that made their way into the repo. And then the agents built up around it.