← Back to context

Comment by limbicsystem

6 years ago

It is true that, as Fisher points out, with enough samples you are almost guaranteed to reject the null hypothesis. That's why we tell students to consider both p values (which you could think of as a form of quality control on the dataset) and variance explained. Loftus and Loftus make the point nicely: p tells you if you have enough samples and any effect to consider, variance explained tells you if it's worth pursuing. Both are useful guides to a thoughtful analysis. In addition, I'd make a case for thinking about the scientific significance and importance of the hypothesis and the Bayesian prior. And to put a positive spin on this, given how easy it is to get small p values, big ones are pretty much a red flag to stop the analysis and go and do something more productive instead.

> "It is true that, as Fisher points out, with enough samples you are almost guaranteed to reject the null hypothesis. "

Where does Fisher point this out?

> "That's why we tell students to consider both p values (which you could think of as a form of quality control on the dataset)"

How is this "quality control"? It just tells you whether your sample size was large enough to pass an arbitrary threshold...