Comment by pegasus
3 hours ago
> you're ignoring the distinct possibility that [...] LLMs could simply stop writing bugs
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
[1] see for example the "Decoherence Trap" article discussed here yesterday: https://maevyn.codes/decoherence
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?
It could lead to a mess, sure. But my favourite prompt on an LLM-untouched codebase is along the lines of "find UB and other bugs. If the fix is less than 5 lines, fix it and commit it. One change per commit". Then I of course check the commit history before pushing.
My experience over the last couple of years with this type of prompt is that eventually you run out of bugs to find (meaning you reach "I see what you mean, but I disagree").
And those fixes don't cause messes.
And a new model may find something new, but not even remotely the volume seen the first time. This indicates to me that the supply of bugs is finite, and after this period of house cleaning, it should go down.
But anyway, a mess (decoherence) doesn't necessarily mean an exploitable mess.
Reading that good article, I find myself thinking that we did not need LLMs to get decoherence in complex software or systems. At least LLMs don't have personal fiefdoms, egos, and company politics adding to encouraging decoherence.