Comment by siliconc0w
8 days ago
Good for them to design and publish this - I doubt you'd see anything like this from the other labs.
The loss of competency seems pretty obvious but it's good to have data. What is also interesting to me is that the AI assisted group accomplished the task a bit faster but it wasn't statistically significant. Which seems to align with other findings that AI can make you 'feel' like you're working faster but that perception isn't always matched by the reality. So you're trading learning and eroding competency for a productivity boost which isn't always there.
It's research from a company that gains from selling said tools they researched. Why does it have to be repeated that this is a massive conflict of interests and until this "research" has been verified multiple times by parties with zero conflict of interests it's best to be highly skeptical of anything it claims?
This is up there with believing tobacco companies health "research" from the 30s, 40s, 50s, 60s, 70s, 80s, and 90s.
I mean, they're literally pointing out the negative effects of AI-assisted coding?
> We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance.
This also echoes other research from a few years ago that had similar findings: https://news.ycombinator.com/item?id=46822158
Dude you falling for so obvious corpo-psyops is so sad. Tobacco companies literally published research that said cigarettes were dangerous too, that didn't stop them from lying to Congress and saying cigarettes weren't totally safe.
Some of you are the reason why there needs to be a new luddite movement (fun fact, the luddites were completely correct in their movements; they fought against oppressive factory owners that treated their fellow humans terrible, smashing the very same machines they used themselves. Entrepreneurs were literally ushering in a new hell on Earth where their factors were killing so many orphans (because many people refused to work in such places originally, until forced by dying in the streets or dying from their labor in such places) they had to ship the bodies of children across towns to not draw suspicion). Until the entrepreneurs started killing them and convincing the king reagent to kill them with the state, they had massive support. Support so high that when suspected luddites were escaping from the "police" you could hear entire towns cheering them on helping them escape).
People rightfully hate this stuff and you refuse to see, the evidence says it's terrible but hey let's still sell it anyway what's the worse that can happen?
1 reply →
I think everyone is aware of this.
But people like that they aren't shying away from negative results and that builds some trust. Though let's not ignore that they're still suggesting AI + manual coding.
But honestly, this sample size is so small that we need larger studies. The results around what is effective and ineffective AI usage is a complete wash with n<8.
Also anyone else feel the paper is a bit sloppy?
I mean there's a bunch of minor things but Figure 17 (first fig in the appendix) is just kinda wild. I mean there's trivial ways to solve the glaring error. The more carefully you look at even just the figures in the paper the more you say "who the fuck wrote this?" I mean like how the fuck do you even generate Figure 12? The numbers align with the grids but boxes are shifted. And Figure 16 has experience levels shuffled for some reason. And then there are a hell of a lot more confusing stuff you'll see if you do more than a glance...
I wish they had attempted to measure product management skill.
My hypothesis is that the AI users gained less in coding skill, but improved in spec/requirement writing skills.
But there’s no data, so it’s just my speculation. Intuitively, I think AI is shifting entry level programmers to focus on expressing requirements clearly, which may not be all that bad of a thing.
> I wish they had attempted to measure product management skill.
We're definitely getting better at writing specs. The issue is the labor bottleneck is competent senior engineers, not juniors, not PMs, not box-and-arrow staff engineers.
> I think AI is shifting entry level programmers to focus on expressing requirements clearly
This is what the TDD advocates were saying years ago.
What AI development has done for my team is the following:
Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments.
Basically all _for free_, while at the same time probably doubling or tripling our pace at closing issues, including some issues in our backlog that had lingered for months because they were annoying and nobody felt like working on them, but were easy for claude to knock out.
I'd be willing to bet that your AI written issues, docs, etc look impressive initially but are extremely low signal to noise. You might be checking some boxes (docstrings, etc) but I do not envy anyone on your team that needs to actually read any of that stuff in the future to solve an actual problem.
1 reply →
I keep describing this as the environmental protection meme, "but what if we make the world a better place - for nothing!"
Even if AI goes away tomorrow, we'll still have better tooling, documentation and processes just because we HAD to implement them to use AIs more efficiently.
> Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments.
> Basically all _for free_
Not for free, the cost is that all of those are now written by AI so not really vetted any longer. Or do you really think your team is just using AI for code?
Interestingly if you look at the breakdown by years of experience, it shows the 1-3 year junior group being faster, 4+ years no difference
I wonder if we're going to have a future where the juniors never gain the skills and experience to work well by themselves, and instead become entirely reliant on AI, assuming that's the only way
I think we're going to see a small minority of juniors who managed to ignore the hype/peer pressure/easy path and actually learned to code have a huge advantage over the others.
Which isn’t saying much if efficiency gains tank the demand for developers, which will then tank everybody’s salary. The actual efficiency gains are debatable, but even if we’re talking about a 20% gain, that could be a few FTEs for a small team.
Anthropic's way into regulatory capture seems to be to pretend they're the benevolent adults in the room. It'll probably work too.
I agree with the Ray Dalio perspective on this. AI is not a creative force. It is only a different form of automation. So, the only value to AI is to get to know your habits. As an example have it write test cases in your code style so you don't have to. That is it.
If you sucked before using AI you are going to suck with AI. The compounded problem there is that you won't see just how bad you suck at what you do, because AI will obscure your perspective through its output, like an echo chamber of stupid. You are just going to suck much faster and feel better about it. Think of it as steroids for Dunning-Kruger.
https://www.youtube.com/shorts/0LeJ6xn35gc
https://www.youtube.com/shorts/vXecG_KajLI
> If you sucked before using AI > you are going to suck with AI
This.
Its 2026 and people still post this. Instead of an upvote?
> The loss of competency seems pretty obvious but it's good to have data
That's not what the study says. It says that most users reflect your statement while there is a smaller % that benefits and learns more and faster.
Generalizations are extremely dangerous.
What the article says simply reflect that most people don't care that much and default to the path of least resistance, which is common every day knowledge, but we very well know this does not apply to everyone.
Relevant quote from their conclusion:
> Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement.
This is very much my experience. AI is incredibly useful as a personal tutor
Yes. I love using AI for the “where do I even start” type questions. The once I’ve had a discussion about various approaches I know what docs to actually look at and I can start thinking about implementation details. I don’t find AI very useful for generating code (weird position I know).
2 replies →
A personal tutor who you remain skeptical of, and constantly try to disprove in order to perfect your understanding.
6 replies →
> there is a smaller % that benefits and learns more and faster
That's not what the study says nor it is capable of credibly making that claim. You are reasoning about individuals in an RCT where subjects did not serve as their own control. The high performers in the treatment group may have done even better had they been in the control and AI is in fact is slowing them down.
You don't know which is true because you can't know because of the study design. This is why we have statistics.
So you don't doubt their conclusion that most sucked by using AI, but you doubt that they found that some learned more?
2 replies →