Comment by jauco

You could repeatedly have developers execute tasks with and without ai (two different tasks. Each subject does the same 2 tasks. Randomize order and what task is done with and what task is done without ai) and show significant differences in duration.

You’d need probably at least 30 to 40 people (there’s ways to estimate this, but this is gut feeling from years ago when I did studies like this)

It would take on a few flaws in tfa which has a low number of people who operate in a codebase that they know very well on issues that they self-selected (and thus already will have a (perhaps subconscious) strategy for).

That would help convincing skeptics.