Comment by limbicsystem
6 years ago
It is true that, as Fisher points out, with enough samples you are almost guaranteed to reject the null hypothesis. That's why we tell students to consider both p values (which you could think of as a form of quality control on the dataset) and variance explained. Loftus and Loftus make the point nicely: p tells you if you have enough samples and any effect to consider, variance explained tells you if it's worth pursuing. Both are useful guides to a thoughtful analysis. In addition, I'd make a case for thinking about the scientific significance and importance of the hypothesis and the Bayesian prior. And to put a positive spin on this, given how easy it is to get small p values, big ones are pretty much a red flag to stop the analysis and go and do something more productive instead.
> "It is true that, as Fisher points out, with enough samples you are almost guaranteed to reject the null hypothesis. "
Where does Fisher point this out?
> "That's why we tell students to consider both p values (which you could think of as a form of quality control on the dataset)"
How is this "quality control"? It just tells you whether your sample size was large enough to pass an arbitrary threshold...
> Where does Fisher point this out?
Probably in the Fisher excerpt.
I looked but did not see it.