Comment by esjeon
8 hours ago
I think the quality of Codex CLI (and Claude Code) is an honest reflection of the status quo of LLM-driven development. The future doesn’t look really promising.
8 hours ago
I think the quality of Codex CLI (and Claude Code) is an honest reflection of the status quo of LLM-driven development. The future doesn’t look really promising.
To be fair we've gone from barely capable at writing code to what we have today in barely 3 years. I wouldn't put money on the counter bet that they won't get beyond this hump.
I wouldn’t assume that progress is linear. Maybe I’m wrong and it will be! But I’m reminded of early developments with self driving cars, it felt like manual driving would be a thing of the past within a decade. But progress has been much slower than that.
> But I’m reminded of early developments with self driving cars, it felt like manual driving would be a thing of the past within a decade. But progress has been much slower than that.
I think that's exactly the standpoint of Yann LeCun.
[flagged]
3 replies →
The product and UX of the tool is distinct from the raw ability of the tool.
Claude code has great UX.
13 replies →
The current approach of the big companies is to keep increasing the number of parameters (there was a report that recent Claude models are on the 4T parameter scale). Even recent SOTA open weight models are in the 1T-2T range.
This is partly due to the multiplicative scaling factor -- the number of tokens; the number of transformer layers; the number of features; etc.
I suspect that for models to have a step upgrade either/both of 1) a new/improved architecture model; and 2) better training of smaller parameter models are needed. I say the latter as I think that with the larger weights there is a lot of unused/redundant parameters.
If we’re learning anything from the past 15 years it’s probably not to extrapolate AI progress in a straight line.
https://finance.yahoo.com/sectors/healthcare/articles/godfat...
Progress is spiky and not guaranteed. The capital costs have also ballooned to the point where the rate has to level out
> The capital costs have also ballooned to the point where the rate has to level out
That's a pretty bold use of "has to". None of the financing behind AI seems sane, yet it continues to defy rational expectations. Reminds me of the comment "the market can remain irrational longer than you can stay solvent". It's not the exact meaning but has the same feeling. The circular financing going on does not help anything either
I dont think the quality has gotten that much better. Take claude 4.5, the probability of getting usable outputs was less likely than with opus 5.5, and the usuable outputs weren't great.
I think newer models are more likely to produce the usuable output, almost always, but its still not great. The frequency in which you get usable results leads one to believe that quality is improving.. No usability is improving but the result is still meh.
Also put an llm in an environment without an lsp or compiler to check against and let me know how capable you think they are on their own. The "loop" is doing a lot of heavy lifting here.
Time doesn't solve bad designs. We still have lots of buffer overflows in C.
We don't see any indication that the LLM companies are slowing down and go for better code.
Instead it just looks like a cash grab until the AI bubble bursts.
Both CLIs are getting worse. I think it's because they keep bolting in unconsidered "features", not because of the code quality.
But maybe being able to code up a feature at the drop of a hat has turbo-charged their ability to make bad UX decisions.
I believe it's all related. They keep adding new, badly thought out features because it's much easier than improving existing features built on low quality code.
The latter requires modifying existing, badly architectured and tightly coupled code that will almost definitely break something somewhere else if touched, so what do you do? Just keep adding new code on top and hope for the best.
This is not a new phenomenon by any means, but AI amplifies it by 100x.
I say this about 30 times a day, to nobody. "How the FUCK is this all so bad? They have unlimited tokens"
I think it’s a new spin on Brook’s Law. Adding tokens to a project makes it less efficient not more.
The "Mythical Man-Month" is now the "Mythical Token-Minute"?
1 reply →
that's what I'm also wondering about.
in the last 2 weeks or so: - scrolling stopped working - subagents no longer work (main agent doesn't get notified when they finish) - subagents don't work (a subagent can't make its own agent even if instructed to do so)
what's going on? I thought coding was solved?
> They have unlimited tokens
That is exactly why it is so bad. Tokens spent generate code. Large amounts of code is a liability.
It has pretty rapidly reached the point where I do not think any acceptable LLM-harness is widely available. Every single one I've tried has broken in bizarre ways on even minor updates.
I used codex the first time yesterday and quickly decided I need to sandbox it and came up with a pretty easy and straightforward solution (I think).
I use nixos and use a little wrapper that wraps Codex CLI in Docker. Run codex-box /path/to/project and it starts or reuses a container with only that project bind-mounted, keeping login and history in a separate volume. It uses tmux for detaching and reattaching
https://gist.github.com/entropie/8f30b9e33e9412fda3adf2f7507...
Build your own, using the broken harness
Maybe thats just the tool nerd in me that just wants the perfect tool for every job.
Very much this... The utility of Claude has grown exponentially with the amount of work I've put into memory/context management, tooling, skills development and CLAUDE.md (and friends). I freely admit I may just be fortunate, but I've yet to hit a bug in a new release that rose above the level of minor annoyance.
The thing that really validated this point was when I recently tried Codex again. Out of the box it was OK, but not great. I swung over the memory apparatus and had it import directives from my existing Claude install and it was almost immediately quite useful in its own right and, especially, as a counterpoint to Claude when dealing with some fairly complex operational issues.
Pi as well? Comparatively it's quite a small surface. I haven't seen issues when I upgrade.
Pi went searching across my entire home directory unprompted... Idk to do that without some kind of sandboxing builtin is rough and chews through tokens on things I don't really want polluting my context.
2 replies →
I find codex CLI to be a pretty high quality application. Is the existence of a backend daemon what you're referring to ?
I've used the CLIs for Codex, Gemini and Claude and they all give off the feeling of being on an very rickety, very fast runaway train.
You install an update. Oops, they added a sandbox that doesn't work inside a docker container, it's stopped working.
You install an update. Oops, you can't log in inside a docker container any more, it's stopped working.
You install an update. Oops, they started catching the copy+paste shortcut and replaced it with one that doesn't work.
You revert a broken update. Oops, the update changed the format of your login details, the old version can't read them, gotta log in again.
You install an update. Oops, can't scroll with the mouse wheel any more.
Not showstopping bugs (if you're used to persuading broken oauth logins to work) but it certainly makes updating an "ugh, what have they broken this time?" experience.
You got further than I did. I checked them out and went back to manual prompting. Just don't have a need for codegen that much, and when I do I'd prefer to be in control.
To be fair, how many products this old even have a revert update mechanism?
It's buggy like hell once you do not run the default settings. Whole permissions system is completely broken, for example.
It’s kinda vibe coding in a nutshell. Lots of cool and varied features, that often just don’t quite work right.
It does some neat things that, ultimately, are super obnoxious. Background animations and "sparkle" effects that make my terminal indicate activity at all times. Just subtly broken behavior.
Totally.
The problem is hardly anyone gives a shit if they can pump out stuff that makes them look like they are good at stuff.
Vanity before engineering.
https://www.jerrypournelle.com/reports/jerryp/iron.html
I think the (lack of) quality of these tools shows that software engineering is maturing as an engineering discipline.
We were overbuilding trivial parts of the software like the Romans overbuilt small bridges.
Modern engineering is about making the thing that barely works. If you disagree, learn a little bit about bit narrow the tolerances of modern cars and bridges are.
> learn a little bit about bit narrow the tolerances of modern cars and bridges are
Buildings typically use a safety factor of 1.5 to 2.0, and cars typically use a safety factor of 2.0 to 3.0. These things are being built twice as strong as they need to be. I wouldn't call that narrow.
And with software, we're at a point where it can't even properly do what it needs to. If we were overbuilding it, damn software wouldn't crash so much.
I think all engineering has safety factors of 2 or more. Maybe with the exception of aerospace (mostly space) where materials might be at the limits of what's possible right now so the safety factors are much reduced.
1 reply →
I think there's a difference between something that's engineered and precise vs something that's rickety. Just because you built something that might fall apart at any moment doesn't mean it's been engineered with narrow tolerances.
I disagree.
> [...] vs something that's rickety
A 100 percent of modern bridges are rickety. There has been dozens of high profile incidents because of this.
> Just because you built something that might fall apart at any moment doesn't mean it's been engineered with narrow tolerances.
Here the software they are building falls apart in ..say.. two days of use, but can be updated in 5 seconds at close to zero marginal cost. The software is useful enough that people keep updating and using it. This iteration gives them a competitive advantage over the safer choice of rigorously testing every new thing for several weeks.
I don't disagree but it's clearly a problem that already existed for decades in software and is just being 10x'ed. how many different frontend frameworks or AWS regurgitated and enshittified cloud services did we have to endure over the years.
There is actually even worse code quality written by humans, is that small sample also a reflection of human-driven development? And therefore, the future doesn't look promising that way either?
The difference is the source of the sample. This is a flagship product by a frontier lab. It should be reflective of the best that is possible. It should be biased in the most favorable possible way. It was designed by people without the normal limitations and quotas of outsiders in building it, as well as in learning experiences in driving it. This is not a random cherry-pick of bad code.
Again, the same parallel can be drawn by frontier tech companies. Case in point: Even before AI (and even till date), iOS has always been buggy, crappy. Apple was a trillion dollar company when they had some of the worst bugs in iOS launched to public.
The same story for Google's products - their products were always buggy even before AI. And so is the case for Microsoft.
I think the reasonable argument is that AI or not, software problems are hard problems to solve. And software is constantly evolving. AI isn't a magic wand. It also depends a lot on the humans using it, including prompting and context supplied to it.
Drawing a massive conclusion from a tiny, unrepresentative sample is just hasty generalization.
Seems like its just a bunch of young engineers making bad architecture decisions. Not sure I would blame the culture of vibe coding everything. It might just be bad taste.
If you blindly trust frontier LLM architecture 'decisions,' they often do look a lot like what a young/junior engineer would do.
The restraint, rigour and responsibility you showed prior to LLMs relates directly to how much of that you apply now that they’re here.
Given how rare it was previously, and how little it was cultivated by shortsighted management, I’m not surprised with all the slop we see today.
The truth is there’s nothing stopping someone from using these tools to remove dead code, simplify project structure, remove dependencies, etc. but I don’t see it happening very often. I have done it a bunch myself though so it’s absolutely possible, it’s all a matter of investment and prioritization.
I'm inclined to agree
i'm personally quite perplexed how sloppy the harnesses are from the leading AI labs, you'd think these are their primary products and would have some human attention devoted to them
it's embarrassing when every claude code release has obvious bugs in it like telling you to upgrade when you're on the latest version or codex opening visible powershell windows for every command that pop up on your screen
like 5 minutes of human testing would catch this obvious shit but obviously they don't bother because vibes are the future and stuff? idk
Solar power has become a real transformative technology and instead of trying to recreate a better world we tariffed them and the execs of this country spent all our coins on power hungry data-centers that write mid code.
I'm sure if you file a bug on the 15k backlog they'll fix it eventually /s
Legit bugs still open 6 months on.. It's a bit ironic