Comment by dcolkitt

IRB (as in Institutional Review Board) is a local (as in each research institution has one) regulatory board that ensures that any research conducted by people employed by the institution follows the federal government's common rule for human subject research. Most institutions receiving federal funding for research activities have to show that the funded work follows common rule guidelines for interaction with human subjects.

It is unlikely that a business conducting A/B testing or a parent interacting with their children are receiving federal funds to support it. Therefore, their work is not subject to IRB review.

Instead, if you are a researcher who is funded by federal funds (even if you are doing work on your own children), you have to receive IRB approval for any work involving human interaction before you start conducting it.

> wouldn’t every single A/B test done by a product team be considered unethical?

Potentially yes, actually.

I still think it should be possible to run some A/B tests, but a lot depends on the underlying motivation. The distance between such tests and malicious psychological manipulation can be very, very small.

> it seems to me this is more about medical experiments

Psychology and sociology are both subject to the IRB as well.

Regardless of their department, this feels like a psychology experiment.

> Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?

I would argue that ordinary A/B tests, by their very nature, are not "experiments" in the sense that restriction is intended for, so there is no reason for them to be considered unethical.

The difference between an A/B test and an actual experiment that should require the subjects' consent is that either of the test conditions, A or B, could have been implemented ordinarily as part of business as usual. In other words, neither A nor B by themselves would need a prior justification as to why they were deployed, and if the reasoning behind either of them was to be disclosed to the subjects, they would find them indistinguishable from any other business decision.

Of course, this argument would not apply if the A/B test involved any sort of artificial inconvenience (e.g. mock errors or delays) applied to either of the test conditions. I only mean A/B tests designed to compare features or behaviours which could both legitimately be considered beneficial, but the business is ignorant of which.

> Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?

Assuming this isn't being asked as a rhetorical question, I think that's exactly what turned the now infamous Facebook A/B test into a perceived unethical mass manipulation of human emotions. A lot of folks are now justifiably upset and skeptical of Facebook (and big tech) as a result.

So to answer your question: yes, if that test moves into territory that would feel like manipulation once the subject is aware of it. Maybe especially so because users are conceivably making a /choice/ to use said product and may switch to an alternative (or simply divest) if trust is lost.

It should be for all science done for the sake of science, not just medical work. When I did experiments that just involved people playing an existing video game I still had to get approval from IRB and warn people of all the risks that playing a game is associated with (like RSI, despite the gameplay lasting < 15 minutes).

Researchers at a company could arguably be deemed as engaging in unethical research and barred from contributing to the scientific community due to unethical behavior. Even doing experiments on your kids may be deemed crossing the line.

The question I have is when does it apply. If you research on your own kids but never publish, is it okay? Does the act of attempting to publish results retroactively make an experiment unethical? I'm not certain these things have been worked out because of how rare people try to publish anything that wasn't part of an official experiment.