Comment by timr

"Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing."

Nobody said that it was. But when you do regular split testing, you can use power analysis to estimate the length of time you need to run an experiment to get a significant result at a certain precision:

http://en.wikipedia.org/wiki/Statistical_power

You can't do this (at least, not easily) when you're using bandit models, because none of the assumptions are valid.