Comment by jefftk
6 hours ago
As is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
The study was updated twice with the last time on August 2026.
The data is still mostly 2025 and the 2026 data is mostly from the first few months, and most production deployments (at least in, say, the Fortune 500) lag the frontier by at least several more months (or much more). I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
> I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
1 reply →
Do you have any references for the models deployed in any of these organizations?
> I suspect it's just not really possible for academic research
But we were told everything changed in fall-winter of 2025 and that when we got “real capable” agents/models. Whatever happened to that?
5 replies →
They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.
By that logic, it is not ever possible to do research on LLM-assisted work.
> The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5.
What we have today is not meaningfully better than what we had a year ago, so that isn't relevant.
Today I can write what I'm looking for, as if I'm writing a design doc, and the models can do the rest. A year ago it was faster for me to just write the code myself. Models today are enormously better than what we had in 2025.
Lol