Comment by status_quo69
2 days ago
No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
30 replies →
> You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
I love this phrase, and I think it captures the difference between the "60 years of legacy code" mentioned elsewhere here, and the vibe-coded monstrosity that we all know we're going to be encountering in the next 5 years. The fact that there were human brains with intent making all the little (short-sighted, myopic, ignorant, etc) changes in the legacy code means that you DO have at least some shot of gaining enough context to make safer changes.
There is no real intent in LLMs and particularly in blind vibe coding, so you'll end up reading stuff that works, or appears to work, but there was no human brain behind the change for you to model and thus predict.
This just means we can't take the (often bad) shortcut of guessing what the code does because we feel we can predict what the author was thinking. Instead, we have to treat existing code as a natural sciences exercise - use the same methods we apply to the world around us, which has its logic and structure, but no human intent behind it.
5 replies →
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
9 replies →
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
2 replies →
> There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug.
At this point, why even bother with source code? Source code is designed for humans to comprehend. If humans are no longer the intended audience, why not have machines output and maintain some more efficient intermediate representation of computer programs?
I fixed many bugs over the last years with help of claude code in an unfamiliar codebase. it is never just Claude alone so we don’t rely on AI understanding 100%. It is whatever AI can understand plus what I do. If you use AI right your understanding will also grow over time.
> But.... That's not knowledge?
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
There are no shortcuts to true understanding.
> wherein the machines are treating them like pets
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
When you get to the point that you are doing 1000 PRs I have to seriously wonder what even are you building, I hear about this a lot from a number of companies, but their “too many commits / too many PRs” problem never seems to give anyone from the outside the feeling they are innovating at groundbreaking speeds.
My favourite quote at the moment: "May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872
I think it's just a different focus on what matters.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
>The software they develop around their model is, relatively, trivial
30,000 changes a day is not my idea of trivial.
I mean a harness around a model is pretty trivial, I’ve also built one (heavily relying on LLMs to build it).
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
5 replies →
I mean, sure we can assume that (although the Anthropic article from the other day say that claude mostly builds Claude) but this is the Claude Code team which, I again am assuming so I could be wrong, focuses on the TUI/CLI interface to those models, not necessarily the models themselves. That coupled with the fact that this is how a large portion of their clientele interact with their moat, it would be odd to me if they didn't apply significant work towards this thing to keep the hype or productivity train moving
> I really don't get the tendency here from Anthropic to describe what's most likely churn? 1000 "PRs" a day on a TUI to wrap a while loop?
From the article, the 30,000 changes a month are in the monorepo, and Claude Code is in a separate repo. So it’s not the TUI that has 1,000 PRs a day.
I could easily see 1,000 PRs a day on the backend. Big tech was doing more than that pre-AI.
When I worked at Facebook (pre LLM) we were around 50-100k per day. I imagine it's much higher now (quantity, probably not quality).
Fair enough, I did invent the 1000 a day because the article was vague, I'll edit my submission accordingly to also be vague
The Anthropic post the other day mentioned part of what helps Claude Code unlock some of this is the sheer ability to use more tokens than the rest to get something going and efficient.
> No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I guess it wouldn't be surprising for people prone to AI psychosis to be attracted to working at the frontier labs, resulting companies that are heavily in the throes of mass AI psychosis.
Being tongue-in-cheek here.
This is a good answer
[dead]