Comment by TheRoque
3 months ago
So with LLM outputting average code, and people using LLM more and more, I guess the average code will become worse over time ?
3 months ago
So with LLM outputting average code, and people using LLM more and more, I guess the average code will become worse over time ?
There is a belief that everyone is just taking whatever the LLM (really agents now) outputs. This is not the case anywhere I work. We use human oversight to have it iteratively improve the code. The average quality is going up.
Do you have some metrics to back this up ? Because from what I can see with my own eyes, outages everywhere, security holes everywhere too, doesn't seem that things are improving..
There are insufficient metrics for both directions, and while you can find evidence to support either position, this discounts two things
1. Things are evolving. The models, and especially the harnesses, are getting better. There was an inflection point at the end of last year, so anything before that is no longer relevant to the discussion. Probably anything before now, since we've had about 6 months with real agentic engineering and things are starting to become clearer.
2. Application and effectiveness is not equally distributed. This is the newest and most significant technology humans have created. We are still building and figuring it out. Some people are better at it, some people use it rather unwisely.
Not advocating for AI code slop--but if AI coded software works correctly, maybe it doesn't matter? Except sometimes when a specialist will have to get involved. Not a perfect analogy, but most people don't write assembly these days--they have a compiler do that. Assembly still has a place, but it's a specialist task.
> if AI coded software works correctly, maybe it doesn't matter?
The problem isn't the amount of code, it's how fitting/unfitting the abstractions are. Wrong abstractions are bugs in waiting. If there's much code with wrong abstractions, future change becomes difficult.
Source: me, I've created many bad abstractions and they led to much pain...
Yeah. Its kind of strange - claude is great at some tasks, but it seems really rubbish at coming up with good abstractions a lot of the time. I've often caught it making a conceptual mistake (like "X cannot do Y") - then spending hundreds of lines working around an issue that doesn't actually exist.
Its also really bad at inventing and leaning on invariants. I make rules in my code all the time - "by the time we get to path X, we know Y and Z are true.". In aggregate, these invariants make code simpler and easier to reason about. But claude doesn't do that. It just kind of - slops through and adds bespoke "just in case" workarounds all over the place. Every time I read through code its written - without fail - I find bad design / architectural choices.
Maybe mythos will change this. But for now I've slowed way down on my claude code usage. You can't build a skyscraper on a foundation of mud.
2 replies →
More verbose code takes up more space in the context. It's harder for humans to review, but also harder for future AIs to edit. Unless you manage to keep the AI to firm module boundaries & have it replace modules wholesale it's not really equivalent to how assembly gets replaced wholesale when a compilation unit changes. Compilers aren't editing the `.o` files when you rebuild, they throw the old ones out & replace them. But when you prompt an AI it is reading & editing the source files, so excess verbosity in the source files is detrimental.
Well, if tokens = cost, and verbosity = more tokens, then smaller code is a financial (and human!) win. Although I'm worried vibe coders are just going to have LLMs modify minified code in caveman mode so they can have 100 agents in a swarm..
On a more serious note, I wonder if this might eventually encourage people to use languages that are a little harder to write but much more concise (functional languages for instance). When you're paying per-token enterprise bean java style verbosity totally sucks
But the truth is: it doesn't work correctly. I see quality of software dropped significantly.
At work we are integrating with third party platform to automate excel-powered calculations. It is awful. Rendering the table in browser takes 10s or one click on Export button will throw backend in OutOfMemory state.
Ai mirrors the code around it. So if there is bad code or good abstractions, it's going to do the same. Even with good code, it will do bad things, you have to remain in the loop and catch these. It can write good code, it just needs nudging.
I don't disagree there is a lot of slop being produced right now, but I'm still optimistic in the long-run.