Comment by PeterCorless

We have a different take on running "one big database." At ScyllaDB we prefer vertical scaling because you get better utilization of all your vCPUs, but we still will keep a replication factor of 3 to ensure that you can maintain [at least] quorum reads and writes.

So we would likely recommend running 3x big servers. For those who want to plan for failure, though, they might prefer to have 6x medium servers, because then the loss of any one means you don't take as much of a "torpedo hit" when any one server goes offline.

So it's a balance. You want to be big, but you don't want to be monolithic. You want an HA architecture so that no one node kills your entire business.

I also suggest that people planning systems create their own "torpedo test." We often benchmark to tell maximal optimum performance, presuming that everything is going to go right.

But people who are concerned about real-world outage planning may want to "torpedo" a node to see how a 2-out-of-3-nodes-up cluster operates, versus a 5-out-of-6-nodes-up cluster.

This is like planning for major jets, to see if you can work with 2 of 3 engines, or 1 of 2.

Obviously, if you have 1 engine, there is nothing you can do if you lose that single point of failure. At that point, you are updating your resume, and checking on the quality of your parachute.