← Back to context

Comment by anthony_doan

6 years ago

I think this article is trying to tie two things together, the p-value problem and the fact you can throw in more data.

I disagree.

It's cheating, it's goes against experimental design analysis, and it does not differentiate between given data and data that was carefully collected. We have experimental design class for a reason. It helps us to be honest. Of course there are tons of pit falls many novice statisticians can do.

It also implicitly leads people to think that statistic can magically handle given data and big data by doing the old fashion statistic way. If you do that than of course you'll get a good p-value.

8 comments

anthony_doan

Reply
gwern  6 years ago

> It's cheating, it's goes against experimental design analysis, and it does not differentiate between given data and data that was carefully collected. We have experimental design class for a reason. It helps us to be honest. Of course there are tons of pit falls many novice statisticians can do.

Explicit sequential testing runs into exactly the same problem. The problem is, the null hypothesis is not true. So no matter whether you use fixed (large) sample sizes or adaptive procedures which can terminate early while still preserving (the irrelevant) nominal false-positive error rates, you will at some sample size reject the null as your power approaches 100%.

  • nonbel  6 years ago

    This is mostly right, but you are still thinking of these rejections as "false positives" for some reason. They are real deviations from the null hypothesis ("true positives"). The problem is the user didn't test the null model they wanted, it is 100% user error.