Comment by throwawaybbq1

4 years ago

This is a good question. You would recruit actual maintainers, [edit: or whoever is your intended subject pool] (who would provide consent, perhaps be compensated for their time). You could then give them a series of patches to approve (some being bug free and others having vulnerabilities).

[edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]

5 comments

throwawaybbq1

Reply
dataflow  4 years ago

If an actual maintainer (i.e. an "insider") approves your bug, then you're not testing the same thing (i.e. the impact an outsider can have), are you?

  • throwawaybbq1  4 years ago

    I meant the same set of subjects they wanted to focus on.

    • dataflow  4 years ago

      How is this supposed to work? Do you trust everyone equally? If I mailed you something (you being the "subject" in this case), would you trust it just as much as if someone in your family gave it to you?

ret2plt  4 years ago

This wouldn't really be representative. If people know they are being tested, they will be much more careful and cautious than when they are doing "business as usual".