Comment by Jabrov
3 days ago
Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
3 days ago
Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.
Did you compare the variance within individuals (due to treatment) to the variance between individuals (due to other stuff)?
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
That's a lot of money for many of us. Do you know those folks were in a HCOL area?
It isn't a lot of money for industry research. Changes of +-40% in productivity are an enormous advantage/disadvantage for a large tech company moving tens of billions of dollars a year in cashflow through a pipeline that their software engineers built.
No idea. They don't say who they were; just random popular GitHub projects.
To be clear it wasn't $75k each.
1 reply →
Neat, how to sign up??
I see these things posted on linkedin. Usually asking $40/hr though. But essentially the same thing as the OP outlines: you do some domain related task assigned either with or without an AI tool. Check linked in. They will have really vague titles like "data scientist" though even though that's not what is being described, its study subject. Maybe set 40/hr as a filter on linkedin and see if you can get a few to come up.
Go back in time, create a popular github repo with lots of stars, be lucky.