Comment by smcleod

Claude 4 Opus when provided with the full paper, blog post and asked to perform a critical review:

The methodology contains several fundamental flaws that likely explain this anomalous result.

Most critically, the study examines a highly specific scenario - expert developers working on codebases they've contributed to for years (averaging 1,500 commits over 5 years) - which creates a ceiling effect where AI has minimal room to provide value.

The 30-minute Cursor training for developers, 56% of whom had never used the tool before, is woefully inadequate for learning effective AI pair programming techniques. With only 16 participants and a non-blinded design where developers knew their condition and were paid $150/hour, the study lacks both statistical power and ecological validity.

The restriction to a single tool configuration (Cursor Pro with specific Claude models) and acknowledgment that the tool doesn't optimise token sampling or prompting strategies further limits the findings' applicability to the broader question of AI's impact on developer productivity.

- Concerningly small sample size: Only 16 developers across 246 tasks provides insufficient statistical power for broad generalisations about AI's impact on millions of developers

- Inadequate AI training: 30-minute basic Cursor tutorial for developers where 56% had never used the tool - completely insufficient for developing effective AI collaboration skills

- Selection bias towards ceiling effects: Developers averaged 5 years and 1,500 commits on their repositories, creating an expertise level where AI assistance has minimal value-add potential

- Single tool learning: Study restricted to education Cursor Pro with specific Claude models, not representative of the diverse AI tooling ecosystem (Windsurf, Cline, Roo Code etc.)

- Artificial task constraints: Tasks were acknowledged as "shorter than average" and broken into ≤2 hour chunks, not representative of real development work

- No experimental blinding: Developers knew their condition and were being observed/recorded, potentially affecting natural work patterns

- Suboptimal AI usage: Study acknowledges Cursor doesn't sample sufficient tokens and developers reported overusing AI due to experimental conditions

- Narrow context: All repositories were large (1.1M LoC average), mature (10 years old), with high quality standards - a specific niche not representative of most development

- Self-reported metrics: Time tracking was self-reported with only 29% verified through screen recordings, introducing measurement bias

- Expertise mismatch: Focusing on experts contradicts established findings that AI tools provide greater benefits to less experienced developers

- Tool proficiency confound: No control for varying levels of AI tool proficiency or different prompting strategies between participants

- Limited generalisability: The specific combination of expert developers + familiar codebases + large repositories + short tasks creates an artificial scenario unlike typical development workflows