Comment by ballsac

2 days ago

> A challenge with this kind of study is that coding agents (Claude Code, OpenAI Codex) only started working really well in

2026? 5? 4? 3?

Heard this one way too many times.

I get the point having read much the same from Tesla (and fans) regarding self driving cars that still haven't done half the things that Musk said was just around the corner pending regulators a decade ago and repeatedly since then.

And myself I keep making comparisons between AI and the progress in 90s video games where every minor improvement got called "photo realistic" and then forgotten with the next game engine: https://archive.org/details/nextgen-issue-26

So I'm not gonna say "this is it" when the software quality really matters, and I absolutely won't speak to progress (or lack of it) outside of software.

But I will say "you can look around and easily see small businesses using AI to generate posters, quite a lot of small business software and websites are in the same category: the mistakes are real but increasingly don't matter".

  • >the mistakes are real but increasingly don't matter...

    I think it would start to matter once again. People will get fed up of AI posters and art. I think they already are...and once some threshold is crossed, the business won't dare to use AI generated assets/designs.

    Turns out humans are much better at recognizing patterns in stuff that is generated ONLY using patterns from human generated content.

    • > People will get fed up of AI posters and art. I think they already are

      Agreed, but will this look like a meme/fashion cycle? If so, re-prompt each year with a different look. Yes, there are still issues here, a friend found an image he was amazed was AI generated, but to me it was obviously so, so I showed him a screenshot of ChatGPT making something just it and included my prompt:

        create image: hand drawing of cute springer spaniel puppy looking sideways, various geometric shapes drawn in layer behind and in front of the puppy, all done in style of 7 year old using crayons with mediocre colouring-in skills
      

      As I said to them:

        yeah, the line thickness feels AI, to me, the bad colouring-in scribbles feel like just the art style it was propmpted with
      
        it's like: it gets the big picture of the composition, and it knows how to colour in badly, but it doesn't know how to draw a dog as badly as the colouring in
      

      > Turns out humans are much better at recognizing patterns in stuff that is generated ONLY using patterns from human generated content.

      We're better at recognising patterns full stop. All biological brains are, and needed to be better than the current state of the art in machine learning because if a living organism was as poor at learning patterns as the SotA in machine learning, the organism would starve to death before being able to pick up anything and eat it.

      AI also has a second disadvantage, because there are so few models: the laziest of ChatGPT "thinkpiece" blog posts being everywhere is hard to miss, and 5000 fake bloggers all prompting the same model with "find biggest news story of today and write a blog post about it in a way that maximises my ad revenue" will get 5000 almost identical posts. This will remain true while each instance of the most commonly used AI fail to talk to each other in a way that at least mimics them collectively getting bored with writing the same thing 5000 times, it does not depend on e.g. quality.

    • It's the exact opposite. Humans will hedonically adapt to anything. If they see enough AI posters pretty soon they will start to not care, at least the majority, while some minority will disagree. We've seen this pattern many times, from data privacy and people still using social media, to using non free software.

      5 replies →

  • I digress with your last quote, I feel like some people who are aware of AI are starting to develop a quasi allergic reaction to slop, and while the mistakes might not matter for most of the population, for others it does and will take notice.

  • I’m probably illustrating your point but as a FSD fan it really got ”good enough” recently with version 14. The tipping point was suddenly, much more often than not, it can drive end to end from start (my garage) to finish (parked at destination) with no interventions. I can text and watch videos on my phone and as long as I glance up once a minute, it doesn’t complain.

    Handling highway driving with lane changes was great when it got there years ago, but just in the last year or so it has gone from a nice to have to “from now on I will never buy a car that can’t do this”.

    AI has hit some milestones for replacing work as well. There’s still many more to go and maybe some of them will never get hit (much like I don’t think a coast to coast drive with zero interventions during winter conditions is ever going to happen) but there are points at which it forever meaningfully changes some field of work. I think it’s there for writing code.

    • > I’m probably illustrating your point but as a FSD fan

      Half-and-half. I'm not denying that self driving cars (and LLMs) are improving, I'm comparing it against the standards set by the biggest proponents. But yes, I have heard basically the same thing you just wrote for the previous several major releases of FSD.

      Where we agree is that, while you are a fan, you do explicitly give as an example of something you think it will never do, something very close to what Musk has promised:

        "Ultimately you'll be able to summon your car anywhere … your car can get to you. I think that within two years, you'll be able to summon your car from across the country. It will meet you wherever your phone is … and it will just automatically charge itself along the entire journey."
      

      - Musk, in Jan 2016: https://en.wikipedia.org/wiki/List_of_predictions_for_autono...

      (That said, I think Tesla's FSD will never get there, not that it's impossible. The way Musk is behaving, there's going to be a financial scheme named after him in whatever passes for a textbook in 20 years, and it won't be the positive kind of example).

      5 replies →

    • I would deeply implore you to not put to much faith in this technology as there are many very dangerous edge cases, including as I personally experienced, jumping red lights after coming to a full stop: https://autos.yahoo.com/policy-and-environment/articles/tesl...

      I keep saying, FSD being marketed as FSD is going to get people killed and I can't believe more is not being done to prevent this.

    • You are going to get someone killed. You should not be allowed on the road

    • Who is responsible for any accident happening while you use your phone during FSD?

      It's not FSD until the human is no longer responsible.

      This half measure bullshit is a joke.

      5 replies →

I guess it's important who one hears this from.

I just spoke to a fried who is a headhunter and who's been trying to automate his processes for a while (he likes to fiddle and certainly has skills, but he's not an engineer). He kept trying, but it just wasn't good enough.

Now he said with GPT Work and Sol, it worked, but the key point is: all of it suddenly worked.

The problem was one of reliability, of handling edge cases. All previous attempts / model-harness-combinations were too brittle and needed too much observation and fiddling - cheaper to do it yourself.

Now he says "I don't know why I would ever hire a recruiter [the folks doing the cold outreach] again. I can focus on the candidate screening and acquiring projects, everything else is fully automated".

This doesn't come from an engineer or an AI lab, but a technically inclined power user, and I think this is where things get interesting.

  • You heard it from someone with no experience developing software. A lot of AI hype comes from that, even (or especially) from people in the actual business of developing software - a surprising amount of managers and adjacent or supporting roles in the field actually have very little clue about software development.

    It's cool that 'regular' people can now create solutions to many small problems, and automate stuff - genuinely a step forward. Like Excel, only vastly better. But for bigger projects, real software engineers know that what LLMs do today is only a tiny part of development. And it solves it in a way that might well make the rest of the lifecycle a lot harder. It's like that saying about tools that make easy things easier and hard things impossible.

    • IMO it doesn’t replace software engineers, but it certainly changes the job. The problem lies with VPs and executives that rose to their positions through managing phones and printers and shit who don’t appreciate all of the work that isn’t “write code”, and those people are the majority outside of big tech in my experience.

      Currently in a re-org justified by AI, AI changing the roles people will need to play. It’s disturbing how much content in the materials about the new org structure and roles and whatnot is clearly ai generated and contradictory. We’re laying off about half of 500 people.

      1 reply →

    • Who are these "real" software engineers and what are they working on? Among software developers I know, the biggest users of LLMs are the seniors. I don't think any of them would agree that its usage is a tiny part of development.

  • > I can focus on the candidate screening and acquiring projects, everything else is fully automated

    would a great candidate get excited about an AI agent reaching out to them? or would it be the desperate or clueless one?

    • If you're a high-end executive, people would not approach you directly but try to find a connection to you (which also serve as references).

      For all other roles the first few messages are similar: this is <role X> with key challenges a,b,c. You seem to be a good fit because of d,e,f. Would you be open to explore this? This requires relocating to <place>.

      Traditionally, this was done by entry-level people. In either case, this isn't the person who will jump on a call with you.

      "Desperate" and "clueless" seem very strong words here in reference to someone who gets actively approached from a recruiter.

  • Then it just becomes a new baseline (everyone have access to the same LLMs), and recruiting moves up the philosophical ladder where human can add more value. What will it be? I don't know, I'm not a recruiter.

  • Again I've heard this since 2022 when gpt3.5 came out.

    This is like microprocessors in the 80s. Sure they double in capability every 18 months but the start is so pathetic it will be 30 years before they are good enough for everyday tasks.

    • CPUs in the 90s were amazing! They were over-specced for "everyday tasks." Our problem is that we overbuilt CPUs too much, so software is now written with ten unnecessary layers of abstraction because there's no real reason to simplify.

      4 replies →

It seems to be true this time though; I have observed it myself and heard it from several experienced developers I personally know and respect. It feels like some threshold was crossed with Opus 4.5 and Gpt 5.3, where the models are now able to reliably solve certain classes of problems that were previously unreliable.

Time will tell of course, and it’s early, but inflection points do exist with progress.

  • Thing is. You can find an extremely similar paragraph written about Claude 4.x or some equivalent gpt. And simultaneously, many people expressing their frustration and the shortcomings of <insert any model>

    “But it’s different this time” - several people, several times over the last couple of years.

    This is not at all a dig at you, I’m very sorry if it reads that way. My point is these things only get truly better in anecdotes. The ways in which they fail is yet to change. Just yesterday I had gpt 5.3 generate completely awful code for the Cinema 4D Python API. Also an anecdote. But for all of the people saying they are truly intelligent and truly reason, they still make obvious mistakes, write around problems, fail entirely at architectural decisions, fail at random, generate FAR too much code.

    And no amount of harnesses, methodologies, loops make much of a difference. If you listen to people on the internet they say it’s all working. You listen to people on the job and they mostly say it’s creating tech debt and a review bottleneck. Also burnout, so much burnout.

    I think LLMs are mediocre. I think it’s fine they’re mediocre. You can work with low expectations. But the hype cycles are so tiresome.

    • I don't buy into the huge LLM hype but I certainly think late 2025 was the inflection point. Up until then, I thought LLMs universally sucked at code. GPT-3.5, GPT-4, o1, o3, Claude 3.5, Sonnet 4, the whole bunch. Each iteration got marginally better, hallucinated APIs less and so on, but my overall evaluation of them all was that they wrote crap code and were unsuitable for anything other than one-off scripts.

      Then the incremental improvements did, in my experience, cross some kind of threshold in late 2025 where the things became useful. It is of course anecdotal and personal judgment. But I asked LLMs to implement a small feature in my codebase (my usual test) and finally it produced code I was happy with. They've also been able to locate and diagnose a problem based on logs. In my view it's now a markedly different level of capability than we had a year ago, though I would call the previous two years equally useless.

    • You can find an extremely similar paragraph written about Claude 4.x or some equivalent gpt

      Yes, as a product gradually improves there will always be many people for whom version X didn't work well for them and version X+1 does. It turns out that Opus 4.5 and GPT 5.1 were larger than average improvements that cross that threshold for a significant number of people.

      My point is these things only get truly better in anecdotes. The ways in which they fail is yet to change.

      If your claim is that there's no substantive difference between Sonnet 3.5 and Fable, then we live in very different worlds.

    • LLM capabilities are spiky. They're amazing at some things, and poor at others. Over time, the set of things they're amazing at has grown, while the set of things they're poor at has shrunk. If you think LLMs are just "mediocre" without any nuance, that's a sign you haven't spent the time to evaluate them in order to make an informed opinion.

    • GPT-5.3 is a last-generation model (which is wild to say when it came out in February this year, but it's true).

      How does GPT-5.6 Sol or Claude Fable 5 or Claude Opus 5 do on that Cinema 4D code?

    • I believe it is a widely accepted opinion that agentic coding took off with Opus 4.5 late in 2025.

      Why would you attempt to use GPT 5.3 to generate code today and form an opinion on that basis?

      I do not think it is even still available in Codex, I believe it only has the smaller, distilled GPT 5.3 Codex Spark.

    • > You can find an extremely similar paragraph written about Claude 4.x or some equivalent gpt

      Even 3.7. I remember when it came out and people were claiming that that was now the model that was going to replace engineers. Cue Fable years later and people still claim that this one is the one.

  • If it's so evident, why can't someone prove it with something more than "it seems better and everyone agrees"?

    • Have you ever seen evidence of a productivity improvement that you have found convincing for any software tool, independent of LLMs?

      What shape did that evidence take?

      1 reply →

    • Con: "Where's the evidence that this actually is improving productivity?"

      Pro: "The evidence hasn't shown up in statistics yet; it's too new!"

      Con: "And won't this destroy maintainability?"

      Pro: "Show me the maintainability disasters caused by AI."

      Con: "I can't yet; it's too new!"

      Both sides are playing the "it's too new" card when asked for actual evidence to prove their claims. In fairness, it actually is too new for there to be much statistically-valid data, especially if the inflection point was November 2025. So both sides are trumpeting their position, neither with actual trustworthy data.

      Everybody has their anecdote. Nobody has data yet.

  • I wonder how much of it is real and how much of it is people just being worn down by the hype to the point they can't fight it anymore

    Very smart people aren't immune to being worn down over time

    • I really don't think that's how it works. Smart, experienced developers who thought coding agents were junk for most of 2025 and think they're useful now in 2026 are not saying that because they got "worn down over time".

      27 replies →

Nobody was saying coding agents started working in 2023 or 2024, because the category was defined by Claude Code which was first released in February 2025.

  • I would say that Aider is what defined coding agents. That was at least multiple months before Claude code. I remember seeing a coworker use aider for a hackathon project Adeline November-December 2024 , and it was already decent and pretty close to the DX we consider coding agents to have

    • Aider was missing a key feature: automatically executing code. That write-execute-fix loop was the big unlock.

      I believe Aider avoided adding that for safety concerns. Claude Code demonstrated that throwing safety to the wing somehow kind of worked out.

    • Devin and Cline were the original true "agents" IIRC. Aider was the original terminal "agent" but it was't a true agent, it was sort of a hybrid agentic chat, since it had limited recursion ability (3 turns by default) and you had to configure it carefully to make it consistently take multiple turns in a row.

Since so many people are doubting you here, here is a post from ~a year ago that's pulling the same "LLMs 6 months ago were crap, now they're awesome" shtick: https://fly.io/blog/youre-all-nuts/. There's more posts along this vein being put out from 2024 on or so.

  • I find this attitude baffling.

    Things are allowed to get better more than once!

    The idea that "yeah, you said technology had improved in the past, and now you're saying it has improved again" is a gotcha just seems incoherent to me.

    • The context for some of these posts are yet-another-null-result coming out, saying that AI isn't having the expected objective result, with the posts defending AI by saying that it totally changed just after the cut-off the research. Evolutionary step changes are definitely a thing, but the problem isn't that AI is having such step changes, it's that you're saying that the most recent step change is the one that shows the improvement, and every study that comes out saying "no, that wasn't," the response to that is, "no, there's a new change that is totally the key inflection point!"

      Or, put differently: if a study comes out next year saying they don't see major impacts from AI in 2026, will you admit your viewpoint as being wrong, or is your response going to be "no, there was a massive inflection point in September 2026 that completely invalidates the paper"?

    • I think people are, rightly or wrongly, identifying goalpost moving re: coding agents are only just this year working well enough to impact work. I'm reminded of a quote I read last year from one of my favorite writers, Noah Smith:

      > The debate over whether AI is taking people’s jobs may or may not last forever. If AI takes a lot of people’s jobs, the debate will end because one side will have clearly won. But if AI doesn’t take a lot of people’s jobs, then the debate will never be resolved, because there will be a bunch of people who will still go around saying that it’s about to take everyone’s job. Sometimes those people will find some subset of workers whose employment prospects are looking weaker than others, and claim that this is the beginning of the great AI job destruction wave. And who will be able to prove them wrong?

      Source: https://www.noahpinion.blog/p/ai-and-jobs-again

      3 replies →

    • I keep running into this as well. Someone shared a link to a study from last year telling me that AI doesn't make programmers as productive as they think. The study was obviously done months prior to it's release. So evaluating the state AI even further back.

      But this didn't seem to concern them. The study said X, therefore it applies to today.

      I'm just not sure it's worth it to argue with others about it at this point. Not that I'm 100% all behind AI coding, but I'm just shocked people are still this resistant.

      2 replies →

  • A year ago it was capable but required much more hand-holding. I thought it was as good as it would ever get and was still happy and productive, but only because I didn't know how much better it would get.

Yep, the goalposts just keep shifting. In reality: they still don't work well, unless you're content with producing low quality work.

  • Your perspective is one that assumes a single entity is using AI, rather than disparate groups of people working on disparate problem spaces/topics, each with different "intelligence" thresholds for them to say "good enough". The goal post isn't changing, it's that there are multiple problem spaces with completely different, fixed, goalposts.

    And even then, within a single group, you'll have multiple thresholds, of "this really helps make my coding more productive" to "I no longer type code, just review" to eventually "I'm no longer employed".

  • This is the opposite of my experience since about February of this year.

    • The quality of the output is so variable. It depends on the model, “effort level”, prompting, probably even the programming language/app functionality, and libraries involved. For example, I find LLMs are best at making simple web apps. These web apps, while simple, would still take a senior engineer perhaps a week or two to create, but LLMs can spit them out inside of an hour. Conversely, LLMs struggle with things like Docker or local model stuff. Parallelization of code is a mixed bag. In these areas I think it often would have been faster for me to write the thing by hand.

      2 replies →

  • "unless you're content with producing low quality work." - With the right guiding hand, it is a productivity multiplier without compromising quality. As a fully autonomous developer, it is a disaster.

    • > With the right guiding hand, it is a productivity multiplier without compromising quality

      This just reads like another variation of “it’s the user not the tool,” which is just endless runway for always blaming people and never acknowledging the limitations of LLM’s.

      I’d be curious to hear how the recipients of your work enabled by the “productivity multiplier” feel about the quality.

      5 replies →

    • How are junior devs becoming qualified “guiding hands” these days? If the expert with LLM assistance is multiplied, what’s a company’s incentive to pay for a junior, and how would they train to get good in these conditions?

      4 replies →

  • Which means good enough for all those companies that outsource their IT, especially for offshoring.

    What they care about isn't software delivery, is physical goods or services that aren't related to software, for them software is a cost center.

  • Here's the rub. They produce low quality work according to your rubric. If that was the universal rubric, they would already have RL'd against it, and you'd like the work they produce.

    • So the average bar for software is very low, and most people writing it don't care about software quality and just want a paycheck to cash?

      I've been saying this for years.

      1 reply →

    • From time to time check a couple of ads in YT. They are full of low quality unrealistic AI generated ads.

  • What kind of work are you doing and what do you consider to be quality or not?

    Of course don’t let me assume, maybe you have a higher quality disproof for the Jacobian conjecture you could share with the class.

The timescale is well established: Late '25 was the start of agentic ai when capabilities of model + api + scaffold reached autonomous state. Any study comapring events before that timeframe is comparing apples with oranges.

Claude 4.5 was it (nov 2025?), without a doubt. It went from frequent hallucinations to highly usable with much less garbage output. If you were making demos of AI tools around this time your demo/pitch/product was saved and you probably looked like a genius.

I haven't. Around the start of 2026 is pretty widely mentioned as when they went from "this is broken slop" to "huh this is actually 90% what I would have written", which matches my experience.

I’ve been feeling gaslit about this too. Getting major “we’re still early!” crypto bro vibes from this constant goalpost moving.

  • The „you will be left behind if you don’t fully embrace the whole thing right now“ is a 1:1 match with cryptocurrency hype

  • The capex is still "early" for sure (i.e. data centers are still being planned and built out, we're hardware/energy constrained).

    If model scaling holds out, we're "early-ish" in terms of the reliability and performance of these systems, just based on utilization of the compute from the planned capex. If we hit hard diminishing returns and we don't find architectural/data workarounds, that would put a wrinkle in things, but I suspect that the AI we have now is capable of helping us find those workarounds and keep things moving.

    • "but I suspect that the AI we have now is capable of helping us find those workarounds and keep things moving."

      lol.

      Anthropic and OAI have unreleased powerful models with no limits - they have yet to pull out a 10-d chess move. Wake up bro. There is no AGI coming.

      1 reply →