Comment by jonahx

I'm guessing you've only ever worked at startups?

Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.

But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.

I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.

Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.

But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.

Here is an explicit example of the additional lack of coherence due to LLMs in the Anthropic Claude Code public repo itself: the human maintainer trusts LLM's incorrect understanding of the current state of code in the repo.

The comment https://github.com/anthropics/claude-code/issues/30407#issue..., written by Claude Code, and posted by the human GitHub account says:

> "Issues here are auto-closed by a bot when they've had no activity for a while and are missing information needed to reproduce (usually a version and steps, or a /feedback ID). Nothing is triaged solely by counting"

In reality all issues in this repo are subject to time-based auto-closing, unless they collect a number of upvotes. See details at https://github.com/anthropics/claude-code/issues/87647.

Therefore, I wonder whether the internal review process described in the Anthropic's blog for internal repos also results in such issues, and whether Anthropic has a internal way to measure and mitigate that. The blog post says:

> "When someone wants to understand a part of the codebase, they ask Claude to write an explainer for exactly that slice. It takes minutes and reflects the code as it is today."

The Claude Code public repo problem shows that LLMs don't always "reflect the code as it is today", and can be a source of additional misunderstanding.

I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.

Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.

What you are saying is leaning on 2 premises that seem to not hold water 1. Code written by people is always more readable and maintainable than LLM code 2. Code written by an LLM is always less readable and maintainable

Can you really make those statements in all intellectual honesty?

> you know it's at least coherent enough that someone understood it

30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.

I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.

GP is saying that "the search space" of "the natural world" is much bigger than the search space of "stuff created on purpose by humans". In both cases, there will be guesswork, but in one case less.

I think there is something to this, but am even more concerned by the pure scale. Consider 100k, 1M, and 10M LOC human legacy projects (already a big problem, usually!), and now compare to the likely outcome of 100M, 1B, 10B, etc LOC llm projects. That's where you may start to see the outcomes I reference in my OP.

You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.

It is not completely useless as a tool but there is still a huge gap in most real-world applications (I work in an org with multiple thousands of devs and I am top 5 AI user consistently, I have used Claude Code since pre-release). If you are building something like an SQL database which has decades of logic tests built by humans then it can work. If you do not have this (i.e. 99.99% of the time) then you still need to be able to issue prompts that give context. And if you start without that context, it will produce shit code in extremely basic scenarios. In particular, humans are far better at producing code that is easy to change (in practice, you need to create unintuitive, non-human rules to govern this i.e. creating many artificial boundaries which would be completely unnecessary for humans).

I am not sure whether the models just capped out in terms of coding or whether they are just trained by people who have never worked on enterprise software but the gap isn't closing. It has just been filled by people who lack the knowledge to identify when they are being bullshitted (the prevalence of consultants suggests that this markets is large).

I have worked on legacy stuff before. In most cases, there is a kernel of reason and logic that can be recovered. I have seen pure AI projects and they are the worst grade of TCS-authored abandonware. They can often do one specific thing but are basically unmaintainable outside of that.

I vibecoded a project without looking at the code as an experiment, and I had the same experience as you. It really drives home the point that, left to their own devices, LLMs produce abysmal code.

If you give them no constraints, they’ll produce code that just works, and it’ll look exactly like that.

We already know the solution to this problem. Software engineering. Put the LLM through the same grinder as any software dev and out constraints into place. Tell the LLM that the code design is bad and that it should improve it.

In my case, I had to go through several rounds of refactoring before the code at least resembled a good codebase.

A fun fact is that the refactorings didn’t break any functionality. Only adding new features broke preexisting behavior.

I'm trying to learn how to write software with agents where I keep control over the design while yielding control over the implementation.

This is not very different from what already happens to architects that delegate implementation to teams.

The challenge Is to find a way to encode the architecture in a way that both you and the agents can understand and agree on.

I'm still learning how to do this effectively, but my initial experience is quite positive. Not because agents are great but because humans are bad; the degree of coherence I've seen agents following is bigger than many human teams I've led in my career.

My benchmark is a quite big project for which the architecture is already well known, namely the reimplementation of the OpenVMS OS (kernel and user-land). There are many traps and many opportunities for the agents to just go ahead and diverge from the goals, so I found it a good playground toearn how reign them in but let them be effective at "drawing the rest of the owl"

i still havent seen claude write code as bad as legacy spaghetti, but i guess im never quite vibe coding since i usually have an idea of the architecture i want, and that leaks into my prompts about how it should approach the problem and verification of the solution?

Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.

So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.

Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.