Comment by lbrito
3 hours ago
>undeniable, massive productivity gains.
How can something so undeniable have zero scientific evidence? Are there any large peer reviewed or meta studies confirming your claim?
3 hours ago
>undeniable, massive productivity gains.
How can something so undeniable have zero scientific evidence? Are there any large peer reviewed or meta studies confirming your claim?
It’s a very hard experiment to run. You have a population that’s already “treated”. You can’t blind them to the fact that they’re using AI tools. It’s hard to imagine a study that wouldn’t have serious flaws that people would then use to dismiss and form their own conclusions. Sure you have METR but that was very low n with a very old model.
I think the surest sign of productivity gains is the sheer volume of adoption. If you look beyond headlines, adoption is just incredible. Of course adoption does not necessarily point to productivity gains, but if this was some sort of FOMO or smoke and mirrors you would not see this much retention and this feverish a pace of adoption. You would not see a large segment of the profession using coding agents exclusively. All of these companies track productivity, again with imperfect proxies, yet everything points to a pretty consistent picture. Same with benchmarks, again a lot of crappy benchmarks but a lot of high quality ones too and a very diverse collection of tasks and capabilities they probe.
Your second paragraph appears to be 3 different instances of saying "X does not necessarily point to productivity gains... but in the case of AI, X definitely means productivity" without really saying why that is true or why other explanations do not fit.
Adoption meaning productivity supposes there are no other dominant factors for the AI push nor AI retention. It is possible for practices to be picked up or continued in spite of causing productivity DROPS. What studies have suggested are factors that make for productive work environments and what is actually enforced in the workplace are different things.
It’s 3 different weak but complimentary proxies. We form beliefs from imperfect evidence and I find these fairly convincing when it’s hard to find any hard evidence of no productivity and exactly the scenario you would expect under the hypothesis that we do see productivity gains. None of this is supposed to be unassailable. I would challenge then if you disagree what the evidence you have for this is?
Adoption implying at least some significant productivity gains doesn’t contradict there being other factors. You’re seeing entire companies reshaped. The argument is this is all for show or CEOs are in some sort of idiot class?
“It is possible for practices to be picked up or continued in spite of causing productivity drops” well of course. I just find that incredibly far away from Occam’s razor.
My point is: we have lots of evidence that’s highly consistent with real productivity gains, and I don’t see many pieces of evidence to the contrary.
Because even in a field like software engineering where the output of our work is save in version control, measuring baseline productivity is hard.
LoC: people argue it’s not what’s important
PRs/day: same as LoC
Getting projects done faster: oh but what about the quality.
Solve the technical problems and actually be more productive, the social systems build around the old way of doing things will hole you back.
Finish a PR in 10 minutes doesn’t matter if you’re waiting days for a human review.