Comment by LeafItAlone
1 day ago
Depends on the prior state.
For one codebase I work in, LLM written code is noticeably and measurably (we have literally measured it for bugs, performance, etc.) better than what existed before. So the LLM code is a welcome change.
Do you measure anything about complexity or brevity? I'm currently working on importing some LLM code and it's 5-10x bigger than it should be for the actual needs.
One of my open questions is how much that increases over time. It could be that is a constant. But it also could be that, say, coding agents will infer false needs from the excess code/docs and elaborate further.
I've only personally utilized it for code _review_ and always ignored the suggested result code instead to understand the source of the issue and try to arrive at a resolution I find satisfactory for my mental model of the issue and solutions I'd prefer.
My take is that the main difference is the approach to problems the coding agents have. They optimize towards presuming a fully working invocation environment, yet checking everything anyway, and then rolling back any changes and re-testing that such changes worked. Which is great for normal types of software but incredibly tedious for anything aimed at less than a fully hands free automated environment.
In other words, AI is very awesome at scope creep of assigned problems and targeting the validation prompts baked into the review system.
It also documents itself extensively
But then you’ll use it to change something, and it won’t update any of the extensive documentation. So then it’s filled with lies.
Or worse
It'll mention the old version then the change, and the relating tickets.
That actually hasn’t been my experience. For all that I dislike about LLM code, it maintains comments well.
Sounds like human generated code and docs. TBH
I fucking hate it.
It encodes context it needs into doc comments and regular comments that make them in insufferable to read and so verbose because the comments contain info on other parts of the code base (that might have changed since that comment was written) but were useful to the model for its implementation. I have so far found no way to stop claude from doing this. It will sometimes do it with hand holding but the moment a task takes a slightly large amount of context its back to the word barf.
I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.
We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.
I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.
I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.
What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.
Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.
What LLM/harness are you using that the results are so terrible?
'ai slop' is hard for me to complain about when it's better than 95% of code prior. The issue is more people trusting it too much to go too far, and you end up with lots of extraneous bullcrap you don't need. Doing too much and too big of a change is still a problem in the agentic world.