Comment by weakfish
4 days ago
> But! Hold my beer… in February 2026 METR effectively walked it back : their follow-up estimates flipped to a speedup (with error bars wide enough to ride a Moto Guzzi, with panniers, through!), and they abandoned the study design entirely - because developers now refuse to work without AI, and can’t reliably self-report time on agentic work. Their latest position: AI probably speeds developers up in 2026, and we can no longer cleanly measure by how much.
This may be true, but they followed in May with this [0]:
> Importantly, survey results are not necessarily grounded in reality. There are reasons to be skeptical of people’s responses to counterfactual questions such as about AI’s effect on productivity — for instance, our study in early 2025 found that people overestimated AI’s effect on their time spent on tasks by 40 percentage points on average.
[0] https://metr.org/blog/2026-05-11-ai-usage-survey/#productivi...
Author here. Thanks, this is a useful addition I'd missed. The 40-point overestimate from their early-2025 work is exactly why I read METR's current position as "we can no longer cleanly measure this" rather than "AI definitely speeds you up now"; self-reports are doing a lot of load-bearing work in every direction, including in my own sense of my productivity.
Good input, and thanks for sharing & writing the article!