Comment by dastbe

consistent hashing requires a bunch of additional memory to solve load variance. there's a nice equation to calculate this somewhere that I'd have to search for, but roughly you need 100 virtual slots in your coordinate space per every destination server to reduce standard deviation to about 10%, and 1000 virtual slots to reduce it to less than 5%.

There's some alternative hashing solutions (rendezvous hash, jump hash, maglev) that improve load distribution without the memory overhead, but still have similar challenges with node addition/removal overhead. Also, integrating state into any of these solutions is incredibly difficult and really antithetical to their point (being deterministic). This is fine when you're assigning uniform workloads and preferred when these assignments need to be long-lived, but for request load balancing you're dealing with uneven workloads and short-lived assignments.

This is where power of two load balancing shines, as it allows you to perform stateful load balancing (using queued requests, cpu utilizations, host perf) efficiently with low memory overhead and minimal effort to add/remove nodes. And if you lose a node with state its no big deal because the lifetime of a request is so short.