Comment by FranOntanaya
3 days ago
This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
If its more "bugs than humans can fix", its also likely "more bugs than humans can triage and verify as legitimate bugs".
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
Not in my experience. I've pointed agents like Claude at existing codebases I work on, and it does a pretty solid job at identifying bugs. It's not garbage or slop at all.
There's a big difference between randos getting agents to hallucinate bug reports on GitHub, vs. when your own internal workflow of LLM-assisted code review is identifying real, legitimate problems in your new features.
This totally matches my experience. It's actually kind of insane how well LLM's like Claude are able to find edge cases, race conditions, overflows, states you didn't think could be reached. It really makes you think about code quality in a whole new way. Becoming aware of so many potential bugs when you write the code, vs. finding out from users over the coming months/years -- it forces you to really consider the costs of quality all at once in a way that simply wasn't possible before.
There’s plenty of things you can very deterministically verify — if the LLM writes a test case which causes an ASan violation, you probably have a bug; if the LLM writes a test case which run the various corruption checking tools and they find corruption then you either have corruption or you have a bug in the corruption checking tool; etc.
There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.
And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.
More bugs than humans can fix, but not more bugs than humans created.
More hypothetical bugs, perhaps?
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
1 reply →
"What even 'IS' a bug really?" HN topics from 2036