← Back to context

Comment by jimbokun

5 days ago

Show me your data.

The only study I’ve seen so far on LLMs and productivity, showed that developers using an LLM were LESS productive than those who didn’t use them.

7 comments

jimbokun

Reply
didibus  4 days ago

There's a dissonance I feel. The study for example looked at experienced developer working on existing open source projects.

Lots of people we're now conversing with could be junior or mid-level, might have tried it for little prototypes/experiments, or for more trivial software like commissioned websites, and so on. They could all be benefiting from agentic coding workflows in ways that we don't. With the caveat that the study you talked about also showed even the experience devs felt more productive, so clearly the use of AI biases your perception of delivery speed.

The large array of context I suspect is responsible for some of that dissonance on online discourse.

ants_everywhere  4 days ago

You could start with a basic literature review.

https://scholar.google.com/scholar?hl=en&q=llm%20developer%2...

> The only study I’ve seen so far on LLMs and productivity

Why are you guys all citing the same study in this thread? There are several studies including a survey paper.

  • didibus  4 days ago

    Personally it's because it's the one that had the best methodology I've seen. Most of the other ones I've seen are kind of a joke. They're mostly self-surveyed, like just asking the participants to say if they were more productive, wrote better code, etc.

    Or you've got some that takes like beginner devs that never worked a job and have them run through some unrealistically simple task.

    Or a handful that are claiming productivity boosts, but they count things like faster than Google search, or fewer keystrokes.

    Or when they measure the "code quality" they just mean the tests passes.

    So when that study came out and I read the methodology, I was like, oh, this is interesting. And then the results were also quite surprising, and surprisingly the one with the best methodology had the opposite finding of all the crappy ones I've seen before.

    Food for thought.