Comment by elcritch
2 hours ago
I read this a lot, do you have examples of this? Last year the code produced by LLMs was pretty sloppy, but recent SOTA models seems pretty good to me. I'm curious if I'm missing something or it's folks using different models or difficulty levels.
From my experience, a snippet of code will look good.
But LLMs seem to have a hard time getting the big picture and reusing code that is already implemented and ALMOST does what you want vs. rewriting everything from scratch.
AI code is extremely difficult to read and follow. It's littered with hundreds of lines of comments and notes, often referring to other notes in other parts of the codebase, and often extremely out-of-date.
Claude seems to me to still focus on getting things done vs. doing it properly, so from my experience it would pretty often mess up your whole codebase just to be able to finish its task, vs. stopping and rethinking the approach.
After a few passes of that, with duplicate code with no underlying coherent vision, hundred of thousands of lines of documentation written as walls of text in markdown and weird coding decisions, your codebase is impossible to work out for a human.
I use Claude Opus 5 at high to plan, Claude Opus 5 at low to execute the plan.
I make it build things in small changes, I give very specific instructions about the architecture, and I make sure to point out existing functionality that can be used instead of writing something from scratch. But it's a lost battle because Claude can't learn and it's basically a black box.
It pretty shitty, but that's what they pay me for so I just do it.
> It's littered with hundreds of lines of comments and notes, often referring to other notes in other parts of the codebase, and often extremely out-of-date.
Adjustable by tooling and prompts - my favorite is a ban on comments >2 lines, a tagged index system for effectively in-repo RAG with short documents on each tagged topic like INFRA-DATABASE-SETUP.md - enforce with tooling that the tag matches the doc and vice versa. Enforce that any PR that has a tag change in it has the relevant doc changes.
> Claude seems to me to still focus on getting things done vs. doing it properly, so from my experience it would pretty often mess up your whole codebase just to be able to finish its task, vs. stopping and rethinking the approach.
Tell it your values - "I value correctness over getting things done, and getting things done properly over speed" solves many of those problems. I've actually spent a portion of today rejiggering my values documents because the models have improved enough I no longer need to be quite so careful about scoping.
I was having those same problems, but particularly using Fable / Sol to do the meta-work has largely eliminated them over the last ~3-4 months. Codebase is now a lean mean token-conserving machine.