Comment by doginasuit
1 day ago
A superhuman working memory is exactly how I've been describing the LLM advantage. Paired with the unreliability of its reasoning and judgment, it is what makes AI a supplement to human intelligence, not a replacement.
On the other side, LLMs make random mistakes and wrong choices and they have a bias toward writing more code instead of less. You can make up for this to some degree by running another LLM against their output, but with very diminishing returns. Even if they were perfect, there will be an ongoing cost to little or no human awareness and understanding of the codebase.
It may take some time for people to recognize the cost of AI code generation and their value for virtually everything else, but I believe we'll get there.
I also find them bad at what I call "abstraction compression." They're really bad at noticing when a helper function is needed, when some structure they repeated five times slightly differently can become a struct, when a whole section of code can be encapsulated in a simpler design.
I'm lucky that for my side project (an interpreter) I've written all the code myself, so I've built up its design in my mind over the past year, and so as I mull over what I'm writing I start coming up with simpler designs.
Interestingly using Opus 5 (and LLMs in general) has made me worse at this, since I don't feel the pain of writing something over and over again. On the other hand, I don't really want to implement a whole stdlib, so I have it write more of the auxiliary code. The hardest thing is that I have to manually manage the context, which is painful when I personally remember every helper function and why. I have to remember to keep the list of helper functions updated, which is irritating.
I have that issue too. One thing I suspect they are amazing at is documentation generation, maybe a good solution is to have it generate a concise summary of helper functions that are available for any given context. Then have it use the same doc when you call in some code generation.
You touched on another baked-in limitation of LLMs: their inability to follow Don't Repeat Yourself. To anthropomorphize a little, LLMs love to repeat themselves. I remember in the early days before reasoning algorithms came along, you could ask an LLM a question and it would often explain the same answer two or three times in a slightly different way. You also see this in image generation with multiple people, they will often do essentially the same person several times with minimal variation. Applied to code, in your functions they saw an earlier pattern and they can't help but write it the same way.
This isn't entirely at odds with producing good code, I often force myself to wait for another occurrence of where a helper function is needed before I write it, lest I create a slew of utilities that are easy to forget about. Maybe it just needs an additional pass: "analyze your output and create a set of helper functions where logic is repeated or the abstraction is wanted."