Comment by timr
11 hours ago
> What do you mean? Contamination and mis-measurement of control samples is a thing that actually happens all the time, and invalidates experiments when discovered.
The entire point of a control is to test for that sort of contamination (or more generally, for malfunctions in the experimental workflow). In the case of a negative control, specifically, you're looking for an "positive" where one should not exist. If an experiment is set up such that you can obtain differential contamination in the controls but not the experimental arms, as you've described, then the entire experiment is invalid.
> What I was trying to say is that if the control is either mis-measured, for example by accidentally counting stearates as microplastics, or contaminated, then the summary outcome may underestimate or understate the prevalence of microplastics in the test sample, even though the measurement over-estimated it.
The control cannot be "mis-measured", any more or less than the other arms can be "mis-measured". You treat them identically, otherwise the control is not a control. Neither example you've given are exceptions: if the assay mistakes chemical B for chemical A, then it will also do so for the non-controls. If the experimental process contaminates the controls, it will also contaminate the non-controls.
What you're missing is that there's no absolute "correct" measurement -- yes, the control may itself be contaminated with something you don't even know about, thus "understating" the absolute measurement of whatever thing you're looking for, but the absolute measurement was never the goal. You're looking for between-group differences, nothing more.
Just to make it clearer, if I were going to run an extremely naïve experiment of this sort (i.e. detection of trace chemical contamination C via super-sensitive assay A) with any hope of validity, I'd want to do multiple replications of a dilution series, each with independent negative and positive controls. I'd then use something like ANOVA to look for significant deviations across the group means. This is like the "science 101" version of the experimental design. Any failure of any control means the experiment goes in the trash. Any "significant" result that doesn't follow the expected dilution series patterns, again, goes in the trash.
(This is, of course, after doing everything you can to mitigate for baseline levels of the contaminant in the lab environment, which is a process that itself probably requires multiple failed iterations of the experiment I just described.)
Most of the plastic contamination papers I have read are far, far from even that naïve baseline.
You’re repeating several of my points in your own words, supporting them and not arguing with them, even though your language and emphasis suggests you think you are arguing.
> then the entire experiment is invalid
Isn’t that what I said? You even quoted me saying it. But I didn’t say anything about only control being contaminated or mis-measured, I think you’re assuming something I didn’t say. Validity is, of course, compromised if the control is compromised, regardless of what happens to the test samples.
> The control cannot be “mis-measured” […] yes, the control may itself be contaminated […]
So which is it? Isn’t the article we’re commenting on talking about the possibility of mis-measuring? Are you suggesting this article cannot possibly be an issue when measuring control samples? Why not?
Controls absolutely can be mis-measured or contaminated or both. It has been known to happen. It’s bad when this happens because it means the experiment has to be re-done.
> If the experimental process contaminates the controls, it will also contaminate the non-controls
Yes! This is exactly what I was implying, and is exactly how you might end up underestimating the relative presence of whatever you’re looking for in the test, if your classification procedure overestimates it.
> You’re looking for between-group differences
Yes! and this is why if, for example, you didn’t notice your control had stearates and you counted them as microplastics accidentally, and then reported that your test sample had 2x more microplastics than your control, you might have missed the fact that your test actually had 10x more microplastics, or that your control actually had none when you thought incorrectly that it had some.
This, of course, is not the only possible outcome, not the only way that the results might be distorted. But this is one possible outcome that the Michigan paper at hand is warning against, no?
> Most of the papers I have read are far, far from even that naïve baseline.
Short of it, or exceeding it? Based on earlier comments, I assume you mean they’re not meeting your standards. I don’t know what you’ve read, and my brief googling did not seem to support your claims here so far. Can you provide some references? It would be especially helpful if you showed recent/modern SOTA papers, work that is considered accurate, and is highly referenced.
> The entire point of a control is to test for that sort of contamination
No, the point of a control is to give you a reference point that shares all the systemic biases and unknown unknowns, not to detect those biases. If you follow the same procedure on a known null and on your experiment and observe an effect, assuming you really did exactly the same thing except the studied intervention, you can subtract out the bias.
This one example of technical jargon diverging from colloquial or intuitive use, and it is the type of thing people who haven't had statistics or scientific process education often struggle with because they keep applying their colloquial intuitions.
You talk like you understand this on the rest of the comment so I'm confused by this framing, and the person you are replying to points out (in my reading ) that contamination of the control 1) does happen in practice (in the sense that there was an accidental intervention) and 2) if the gloves contaminated both the measurements and control the same way then the control is exactly serving it's purposes