Comment by capableweb
4 years ago
> It seems that the research in this paper has been done properly.
How is wasting the time of maintainers of one of the most popular open source project "done properly"?
Also, someone correct me if I'm wrong, but I think if you do experiments that involve other humans, you need to have their consent _before_ starting the experiment, otherwise you're breaking a bunch of rules around ethics.
They answer to this objection as well. Same section:
> Honoring maintainer efforts. The OSS communities are understaffed, and maintainers are mainly volunteers. We respect OSS volunteers and honor their efforts. Unfortunately, this experiment will take certain time of maintainers in reviewing the patches. To minimize the efforts, (1) we make the minor patches as simple as possible (all of the three patches are less than 5 lines of code changes); (2) we find three real minor issues (i.e., missing an error message, a memory leak, and a refcount bug), and our patches will ultimately contribute to fixing them.
And, coming to ethics:
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
> They answer to this objection as well. Same section:
Not sure how that passage justifies wasting the time of these people working on the kernel. Because the issues they pretend to fix are real issues and once their research is done, they also submit the fixes? What about the patches they submitted (like https://lore.kernel.org/linux-nfs/20210407001658.2208535-1-p...) that didn't make any sense and didn't actually change anything?
> And, coming to ethics:
So it seems that they didn't even just mislead the developers of the kernel, but they also misled the IRB board, as they would never approve it without getting consent from the developers since they are experimenting on humans and that requires consent.
Even in the section you put above, they even confess they need to interact with the developers ("this experiment will take certain time of maintainers in reviewing the patches"), so how can they be IRB-exempt?
The closer you look, the more sour this whole thing smells.
> The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter.
I was wondering why he banned the whole university and not just these particular researchers. I think your quote is the answer to that. I'm not sure on what basis this exemption was granted.
Here's what the NIH says about it:
Definition of Human Subjects Research
https://grants.nih.gov/policy/humansubjects/research.htm
Decision Tool: Am I Doing Human Subjects Research?
https://grants.nih.gov/policy/humansubjects/hs-decision.htm
And even if they did find some way to justify it under their own rules, some of the research subjects clearly disagree.
Because in the paper is stated that they used partially fantasy names. So far they've found only 4 names of real @umn.edu people from Kangjie Lu's lab, which could easily be blocked, the most coming from two of his students, Aditya Pakki, Qiushi Wu, plus his colleague Wenwen Wang. The Wenwen Wang fixes look like actual fixes though, not malicious. Some of Lu's earlier patches also look good.
https://lore.kernel.org/lkml/20210421130105.1226686-8-gregkh... for the full list
Is "we acknowledge that this will waste their time but we're going to do it anyway" really an adequate answer to that objection?
They appear to have told the IRB they weren't experimenting on humans, but that doesn't make sense to me given that the reaction of the maintainers is precisely what they were looking at.
Inasmuch as the IRB marked this as "not human research" they appear to have erred.
Sounds like the IRB may need to update their ethical standards then. Pointing to the IRB exemption doesn't necessarily make it fine, it could just mean the IRB has outdated ethical standards when it comes to research with comp sci implications.
It doesn't make it fine, no. But it does make a massive difference — It's the difference between being completely reckless about this and asking for at least token external validation.
1 reply →
At least one human, GKH, disagrees.
To me, this further emphasizes the idea that Academia has some serious issues. If some academic institution wasted even 10 minutes of my time without my consent, I'd have a bad taste in my mouth about them for a long time. Time is money, and if volunteers believe their time is being wasted, they will cease to be volunteers, which then effects a much larger ecosystem.