Comment by viraptor
2 days ago
The good old regression to the mean. Testing models as the second pair of eyes only when the first fails is going to give weird results... https://www.smbc-comics.com/comic/protocol
2 days ago
The good old regression to the mean. Testing models as the second pair of eyes only when the first fails is going to give weird results... https://www.smbc-comics.com/comic/protocol
How does the SMBC comic demonstrate regression to the mean? Seems like it's a better example of a post-hoc fallacy.
If you choose the worst performers, a part of their failure will be due to luck and part due to skill. Letting them retry will average the luck in that sample. (So likely increase) Of course you as the tester could be very unlucky and pull a sample of people with no skill who did get a bit lucky in the first place, but it's both unlikely and... it's a funny comic.