Comment by spenczar5
2 years ago
This is a fun article. If I could make one suggestion to the author, it would be to do away with the p-value, and use a more sophisticated measure, like bootstrap resampling differences between the control and test distributions. You would get direct characterization of the distribution of the difference of the mean, and could present the full distribution or confidence intervals or whatever. Just a lot more useful than the crummy KS test.
Explaining and utilizing bootstrapping would make this post even longer and much more difficult to understand for non-statisticians.
Bootstrapping is best used for compensating for low amounts of data, which is why I suggested a change going to forward is to generate much more synthetic data.
Would it? You didnt need to explain the theory behind the KS test. The result is easier to interpret - it could be something like “the $500 tip results in answers that are 0.95 characters closer to the target, on average”. That seems a lot better than the unitless, weirdly scaled KS values.
Bootstrapping works great for any volume of data. Its also nice that mean-difference bootstraps have extremely few distributional assumptions, which is really handy with these unmodelable source data distributions.