Comment by RosanaAnaDana

I think a major issue here is that, perhaps, there is a tendency to want to use statistics to decide what the 'truth' is, because it takes the onus of responsibility for making a mistake away from the interpreter. Its nice to be able to stand behind a p-value and not be accountable for whatever argument is being made. But the issue here, is that most any argument can be made in a large enough dataset, and a careful analyst will find significance.

This is of course the case only if one does not venture far from the principal assumptions of frequentism, most of which are routinely violated outside of almost every example except pure random number generation and fundamental quantum physics.

So a central issue that isn't addressed in STATS101 level hypothesis testing is the impact that the question has on the result. Its almost inevitable that people want to interpret a failure to reject as a positive result. But a p-value really doesn't tell you if its a useful result; but rather, your sample size is big enough to detect a difference.

Statistical significance is something that can be calculated. Practical significance is something that needs to be interpreted.