Comment by aerhardt
1 day ago
I find this analysis confusing. PMF for coding was likely reached some time last year. Profitability, which is different, we don’t know. The article kind of confuses both without making a strong economic case or using numbers in a compelling way. I don’t understand what the Uber case has to do with this either. The Uber COO clearly said that at least in terms of ROI he’s not seeing the results either.
My take is the product has been very useful for coding (PMF) for months. But it’s certainly not useful at any cost…
What I also find confusing though is that folks seem to ignore trajectory which is maybe the biggest lede to bury. As Simon says, we have had "good enough" coding agents for 6 months, that is a blink of an eye, and at my company my job has now completely changed. It's almost like a dream.
And that's just one inflection point. We've had several and there are many more on the horizon. So while I could be convinced that ROI is maybe not even positive today despite the ridiculous enterprise spend, it's perfectly rational to pave the way today for what's coming over the next few months let alone years down the line.
There may be additional major leaps forward, and there may not. I kind of struggle to imagine what the next step actually is. Certainly there will be improvements in performance (speed) and cost. But at a point you reach a barrier where the limiting factor is the specificity of the human prompt and our ability to manage all the code we’re generating.
Somewhat oversimplifying; writing software and building apps was a bottleneck - now it is not. What is the next bottleneck that LLMs can solve? Is there one? And is there enough publicly available data to solve it repeatably at scale? Or did we just automate stack overflow searches and now we’re stuck again?
Or is the endgame of this innovation cycle the complete removal of interaction with machines through code? Will we simply interact with machine coworkers purely through natural language? Can an LLM make PowerPoint slides and run a meeting? So far not seeing much progress on that.
Based on how much money is chasing returns, and how steep the slope is, it's almost certain that we are still not at the end of this sigmoid cycle.
Sure, it might start to slow down, but even then we will likely see a doubling in the next 10-15 years.
https://substackcdn.com/image/fetch/$s_!_ZW2!,f_auto,q_auto:...
Judging from the fact that the Opus 4.5 inflection point was not really anticipated, and we still don’t really know what threshold was crossed that suddenly made agentic coding accessible to so many more people, I think it’s safe to say we don’t know what the thresholds will be until they’re crossed. The fact that we don’t know exactly what they’ll be isn’t a good reason to think there won’t be any more.
4 replies →
I am currently eating lunch. Meanwhile Claude is triaging and writing reproducers for 70+ tickets nobody has had time to look at. Next it will attempt to fix them. I have not read the tickets. I will not look at the code until there are review ready PRs and a code review bot have done the first pass.
In other words, most of the prompting will also go away.
2 replies →
yeah but if you have to pay $2k to $3k per month, would you still use it?
PMF implies profitability. I could give away dollars for $0.80 and have unlimited demand but it doesn't mean I've found PMF.
Pmf is this weirdly defined thing where "if you're not sure you have it then you don't".
I think it was clearly useful for months to people who had tried it and taken the time to understand it, but now that knowledge has spread to the point where wallet holders are convinced it's not just passing fad or hype so now pmf can be "claimed".
I agree it's weird to say "those people have pmf" though, usually it's something you define for yourself
> Pmf is this weirdly defined thing where "if you're not sure you have it then you don't".
I'm not sure if this runs counter to your point or not, but: I don't see any future where LLMs aren't a core part of Software Engineering. The horse is out of the barn. There is no going back.
Yeah but the product is not “LLM” it’s “proprietary frontier model LLM paid by the token”.
And I don’t even necessarily disagree with OP! It’s more like the competition is shifting so quickly that your competitors could undercut your PMF in a blink of an eye.
3 replies →
True but that is maybe 5% of what is being promised by the average booster
1 reply →
> clearly useful for people who took the time to understand it
people -> programmers, I haven’t met a non-developer who reports getting more time out of current AI platforms than they put in. If anything I’ve anecdotally heard the opposite, introducing AI at work creates so much slop (output) it takes more time to process it all without a tangible bump in overall productivity
I have at least a half dozen examples of people not hiring people or buying other tools/subscriptions because they built their own with Claude
The article also treats the word "good" as load-bearing in a way that should have you questioning their analysis:
"I’ve called November 2025 the November inflection point because that was when GPT-5.1 and Opus 4.5, combined with their respective coding agent harnesses, got good—good enough that we’ve spent the last six months adapting to agent systems that can reliably get useful work done."
Yet it’s backed up by adoption across the industry
MongoDB was once backed up by adoption across the industry. Or for a more recent example, blockchain took off like wildfire across the industry before ultimately fizzling out in all but the most niche applications.
Not saying this trend will do the same, just that the industry adopting something doesn't guarantee its success.
2 replies →
Correct the cost is part of the economics.
Thats why most here shouldn’t engage in the discussion - they parrot on about benefits without identifying and articulating the costs and moreover how it affects the firms financial position.
And what happens when open models catch up in 6 months or so?
It’s not supposed to be logical, it’s an LLM evangelism blog that rarely, if ever, has any critical analysis that isn’t pro-industry. Read any/all of the other posts and you won’t find much skepticism but you will find a lot of shilling how great it all is.
I like his other posts. He's bullish on AI, which is fine. I'd like to read a mix of bearish and bullish level-headed takes from people who are subject matter experts. His technical credentials are well past discussion - I love Django, and he comes across as a pretty upbeat but level-headed guy. Certainly beats radical takes in either direction from people who have no clue what they're talking about. It's just this article that I find rather confusing.
The thing that matters most to me is if reading what I wrote teaches you some new things and gives you something useful to think about.
If I make an argument and you disagree that's fine with me, provided I didn't use misinformation or sloppy thinking in making that argument.
2 replies →
308 posts on AI ethics: https://simonwillison.net/tags/ai-ethics/
52 on AI misuse: https://simonwillison.net/tags/ai-misuse/
149 on the unsolved challenge of prompt injection: https://simonwillison.net/tags/prompt-injection/
40 on slop: https://simonwillison.net/tags/slop/
If you want an "LLM evangelism blog that rarely, if ever, has any critical analysis that isn’t pro-industry" there are plenty out there. I'm not one of them.
People are confusing "excitement" with "evangelism". Your blog is definitely on the pro-AI side of things, but as you say, it's not one-sided or uncritical.
No offense, I’ve read nearly all of your posts and the criticisms of LLMs are still pretty heavily steeped in enthusiasm. And a lot of your short “criticism” pieces are nothingburgers other than detailing some incident (which I would categorize as blogspam).
I think you should highlight your exemplary pre-AI writing too.
All of these are about AI misuse, not skepticism of AI. By skepticism I mean doubting whether AI actually delivers on its promises which, based on this last post, sounds like something you think we're already past.
Many people still think AI coding agents are slop on steroids despite all the current hype around AI actually shipping functional products.
14 replies →