Comment by nfw2

LLMs don't seem to be really that good at managing complex and conflicting needs, especially if it grows in increments, a ticket here and there. (To be fair people often suck at it too.)

I have a theory that mess in an LLM-generated codebase sort of grows exponentially. Each hack or sloppy abstraction chips away at the coherence of the project as a whole and makes the next feature harder to do. But perhaps more importantly, it demonstrates to the LLM, which is a probabilistic machine despite its impressive capabilities, that quality of the code is low, and it generates more low-quality hacks.

And eventually you reach knots upon knots that you have to untangle yourself because the LLM becomes too confused by the code it wrote itself. As models get better, the problem seems to be getting better. I'm needing to do this more and more rarely, and the tradeoff is still clearly worth it.

But I've seen Fable 5.1 absolutely lobotomized by someone who fills each chat context with half a million tokens of insipid garbage, and I don't see why this couldn't happen when coding as well. I'm not sure a solution to this general problem that LLMs can simulate idiots really well will be fixed any time soon.