Comment by kridsdale1
6 days ago
The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.
6 days ago
The systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.
To make sure user id U doesn’t always end up in eg control group it’s useful to concatenate the id with experiment uuid.
How do you handle different users having different numbers of trials when calculating the "click through rate" described in the article?
careful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production
The user ID is non uniform after hash and mod? How?
If you mod by anything other than a power of two, it won't be. https://lemire.me/blog/2019/06/06/nearly-divisionless-random...
2 replies →
additional to the other excellent comments they will become non-uniform once you start deleting records. that will break all hopes you might have had in modulo and percentages being reliable partitions because the "holes" in your ID space could be maximally bad for whatever usecase you thought up.
Just make sure you do the hash right so you don’t end up with cursed user IDs like EverQuest.