← Back to context

Comment by ahachete

1 month ago

Postgres can really scale well vertically (and horizontally for read-only workloads) as the post shows.

However, I'm still surprised about the reasons for not sharding. They have been mentioned before, but I haven't seen a substantial rationale.

Sharding is almost only analyzed from the perspective of write scaling. But sharding may not only be about write scaling, but a path to reducing blast radius. And this is something that triggers much earlier than write scaling needs (especially given how well Postgres scales vertically and reads).

When you shard your database, you end up having N clusters (for HA purposes, each "shard" must be a primary-replica(s) cluster itself), each holding 1/Nth of the data.

There are certain scenarios which, while unlikely, may hit you: data corruption in the WAL replication stream, a problem during a major upgrade, a situation that requires a whole cluster restore from a backup, you name it. For those cases, the whole cluster may experience notable downtime.

If you have a single cluster, 100% of your users experience downtime. If you sharded into N clusters, only 1/Nth of your users experience downtime. For a company servicing 800M users the difference from both scenarios is dramatically different. Even for much much smaller companies.

I'm puzzled why this is not perceived as a risk, and if it is not, how it is mitigated.

While I wouldn't advocate to shard "too early", given that it comes with notable caveats, I believe more and more in sharding your workloads when possible more earlier than later. Way before truly needing it from a write scaling perspective. Because apart from reducing the blast radius, it applies implicitly the principle of "divide-and-conquer", and your problems become much more manageable (your number of connections per cluster decreases at will, backup restore times can be a fraction of the time, logical replication can be considered as a true option for replication/upgrades/etc if you keep shards relatively small and many other operational procedures are greatly simplified if now you have much smaller databases, even if you have many more of them).

Microsoft originally bought CitusData and rebadged it as Azure CosmosDb for Postgres Cluster. Microsoft have been recommending partners to now avoid that product. It does not and will not support Entra federated workload identities (passwordless).

The replacement will be Azure Database for Postgres with Elastic Clusters. I think it is still in preview.

Again it’s Citus based, but without the CosmosDb badge and it will support federated workload identities.

https://techcommunity.microsoft.com/blog/adforpostgresql/pos...

https://learn.microsoft.com/en-us/azure/postgresql/elastic-c...