Comment by wahnfrieden
2 days ago
That study was not conducted well at all. The participants haven’t learned how to use these tools. For example one was interviewed later and said a lot of the time they would wait for an agent and get distracted playing with something irrelevant and then forget to go back until much later. That has solutions they are not aware of to implement.
> For example one was interviewed later and said a lot of the time they would wait for an agent and get distracted playing with something irrelevant and then forget to go back until much later
Counterpoint: the agents are the reason for the distraction.
> That has solutions they are not aware of to implement.
Counterpoint: there are no other current studies that suggest otherwise. Given the impact of LLMs on open source (net negative, maintainers are drowning in slop: e.g curl: https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-s...) maybe it makes sense to be a bit more critical on LLM's supposed gains.
Let's see what we have so far:
- The only study to date suggests a net negative from using LLMs on experienced developers
- OSS maintainers are rejecting AI generated PRs due to low quality
- No other studies have come out so far to suggest otherwise
Based on my anecdotal experience and based on the currently available evidence, for me the conclusion is clear: LLMs and agents are mostly hype.
You keep talking about "no other studies" as if that holds power but the strength of your argument rests on a single study
It's no surprise to me that devs who are accustomed to working on one thing at a time due to fast feedback loops have not learned to adapt to paralellizing their work (something that has been demonized at agile style organizations) and sit and wait on agents and start watching YouTube instead. The study reflects usage of emergent tools without training, and with regressive training on previous generation sequential processes, so I would expect these results. If there is any merit in coordinating multiple agents on slower feedback work, this study would not find it.