Comment by cyb_
8 hours ago
I would reframe this as: Quality was always a significant bottleneck and is even more so with agentic coding.
Code review is where many of the design & code quality problems are observed/highlighted.
8 hours ago
I would reframe this as: Quality was always a significant bottleneck and is even more so with agentic coding.
Code review is where many of the design & code quality problems are observed/highlighted.
And test creation and maintenance. AI-generated tests will often embed existing bugs as testable patterns that fail when fixed or rely on empty asserts to increase coverage. A human author generally knows better. Coverage is increasingly becoming theater and you have runtime testing eating up CI resources and time with bloated, unnecessary, untargeted testslop that haven't been mutation tested or a human eye assessing if the business logic is actually represented.
It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.
So far this year, of the defects severe enough to warrant a formal RCA process that I’ve dealt with, “AI generated tests asserted that the defective behavior was correct,” was a contributing factor in every single one.
I haven’t had this happen yet, but I was just thinking about this last night. I would be very interested to see if exercising TDD, as difficult as that practice is to actually do, would have a meaningful positive impact on improving this phenomenon. Usually agents model what most humans do, writing tests to pass and assert that the code you just wrote works as you wrote it.
2 replies →
This. The ability to know that a vibe coded PR is all right to merge, is a function of both the thoughtfulness and simplicity of the underlying architecture, and the level of ownership that contributors have of the underlying products. Both benefit more nimble teams.
Agentic code review can also be employed, but it's worth noting that, as the so-called "Bitcoin Red Team" (which has been testing all of Bitcoin related software in the wake of the July 30 Coldcard attack) has discovered a very useful thing to consider: American models are absolute trash when it comes to reviewing code with an eye on security, due to their "safeguards." This applies even when the code being scanned for vulnerabilities was just written by the same agent.
Kimi K3 seems to be the favorite there, though I believe some of the other open weight/open source models are being employed a bit as well.
And yes, even that requires human oversight -- but it does a quick rough filter through things so you're not wasting time on the little stuff, allowing you more attention to find the glaring holes.