Comment by wahnfrieden
1 day ago
That study was not conducted well at all. The participants haven’t learned how to use these tools. For example one was interviewed later and said a lot of the time they would wait for an agent and get distracted playing with something irrelevant and then forget to go back until much later. That has solutions they are not aware of to implement.
> For example one was interviewed later and said a lot of the time they would wait for an agent and get distracted playing with something irrelevant and then forget to go back until much later
Counterpoint: the agents are the reason for the distraction.
> That has solutions they are not aware of to implement.
Counterpoint: there are no other current studies that suggest otherwise. Given the impact of LLMs on open source (net negative, maintainers are drowning in slop: e.g curl: https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-s...) maybe it makes sense to be a bit more critical on LLM's supposed gains.
Let's see what we have so far:
- The only study to date suggests a net negative from using LLMs on experienced developers
- OSS maintainers are rejecting AI generated PRs due to low quality
- No other studies have come out so far to suggest otherwise
Based on my anecdotal experience and based on the currently available evidence, for me the conclusion is clear: LLMs and agents are mostly hype.