Comment by peab
2 days ago
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.
It is not completely useless as a tool but there is still a huge gap in most real-world applications (I work in an org with multiple thousands of devs and I am top 5 AI user consistently, I have used Claude Code since pre-release). If you are building something like an SQL database which has decades of logic tests built by humans then it can work. If you do not have this (i.e. 99.99% of the time) then you still need to be able to issue prompts that give context. And if you start without that context, it will produce shit code in extremely basic scenarios. In particular, humans are far better at producing code that is easy to change (in practice, you need to create unintuitive, non-human rules to govern this i.e. creating many artificial boundaries which would be completely unnecessary for humans).
I am not sure whether the models just capped out in terms of coding or whether they are just trained by people who have never worked on enterprise software but the gap isn't closing. It has just been filled by people who lack the knowledge to identify when they are being bullshitted (the prevalence of consultants suggests that this markets is large).
I have worked on legacy stuff before. In most cases, there is a kernel of reason and logic that can be recovered. I have seen pure AI projects and they are the worst grade of TCS-authored abandonware. They can often do one specific thing but are basically unmaintainable outside of that.
> You can tell with some of the more recent models that they have been fine-tuned in such a way to produce a lot of descriptions that sound impressive but have no real meaning. In terms of production of semantic text, the recent models are significantly worse than a year ago.
I frequently find this in LLM generated comments, both when using an LLM myself or when reviewing code that someone else used an LLM for. I find it is often the most obvious indication that nobody really read through what was generated.
Yes, it has become quite common that Claude’s text feedback, e.g. Claude found an issue in the existing code that requires a decision and Claude tries to explain it, it is a paragraph of bullshit words stringed together.
Then you investigate it and it is often something relatively simple that could have been explained with short sentence.
1 reply →
We had one that hallucinated benchmarks, with graphs and everything! (The code didn't actually work.)
I vibecoded a project without looking at the code as an experiment, and I had the same experience as you. It really drives home the point that, left to their own devices, LLMs produce abysmal code.
If you give them no constraints, they’ll produce code that just works, and it’ll look exactly like that.
We already know the solution to this problem. Software engineering. Put the LLM through the same grinder as any software dev and out constraints into place. Tell the LLM that the code design is bad and that it should improve it.
In my case, I had to go through several rounds of refactoring before the code at least resembled a good codebase.
A fun fact is that the refactorings didn’t break any functionality. Only adding new features broke preexisting behavior.
I don't think this is the case anymore if you're using the latest expensive models, i.e fable.
I'm trying to learn how to write software with agents where I keep control over the design while yielding control over the implementation.
This is not very different from what already happens to architects that delegate implementation to teams.
The challenge Is to find a way to encode the architecture in a way that both you and the agents can understand and agree on.
I'm still learning how to do this effectively, but my initial experience is quite positive. Not because agents are great but because humans are bad; the degree of coherence I've seen agents following is bigger than many human teams I've led in my career.
My benchmark is a quite big project for which the architecture is already well known, namely the reimplementation of the OpenVMS OS (kernel and user-land). There are many traps and many opportunities for the agents to just go ahead and diverge from the goals, so I found it a good playground toearn how reign them in but let them be effective at "drawing the rest of the owl"
i still havent seen claude write code as bad as legacy spaghetti, but i guess im never quite vibe coding since i usually have an idea of the architecture i want, and that leaks into my prompts about how it should approach the problem and verification of the solution?
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
One problem is managing expectations from non-technical people. You can ship faster but you are also producing tech debt at that same faster rate, and possibly even faster as the gaps in your initial implementation/prompts are filled by Claude deciding to do random shit.
In most orgs, there is also a gap between leadership, even tech leadership and people actually using LLMs every day. And it is challenging for actual devs who are being told that they should be going much faster and the reality which is more mixed. Pushing back on people who are convinced that Claude is God is going to be tough if you are a junior.
I don't feel the job has really changed, I don't write code as much but that time goes into working out the structure...which is what I was doing anyway. But it has changed the general environment, how work is planned, expectations, etc.