Comment by m11a
3 hours ago
This article is a bit hard for me to grasp the main ideas of because, given Cloudflare's requirements (e.g. no strong leaders), it immediately seems like they should be comparing to leaderless protocols like Paxos-class algorithms. Comparing to Raft and saying it's better because Meerkat is leaderless is confusing, because Raft is an adjustment to Paxos to specifically have strong leaders. So I'm 3/4 the way into the article and I don't see what's unique here.
I think the unique idea here is supposed to be QuePaxa's idea of avoiding timeouts for ensuring liveness. The actual discussion of QuePaxa is limited to one paragraph at the end, and tbh only a couple sentences of that paragraph.
I feel like the article could've been titled "Consensus protocols and linearizability: a brief explainer", or "Paxos vs Raft", or similar. It just doesn't feel like it communicates what it claims to communicate, and is a bit confused on who its audience is, just IMO.
Worth noting if it wasn't obvious from the article that Cloudflare did not develop QuePaxa. It's from an SOSP paper back in 2023[0]. The article is discussing what is the first known large-scale public deployment of the protocol.
[0] https://dl.acm.org/doi/10.1145/3600006.3613150
Right. But given that the entire point revolves around QuePaxa, it's strange to see no discussion on it. If that weren't the point, the article would be "Why Cloudflare implemented and deployed Paxos".
Which would also be a good read, but this article also isn't that. It doesn't discuss their experience deploying the protocol, aside from the following statement:
> Meerkat is not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate.
I think it would've been more interesting to read why Cloudflare chose the specific algorithm they did, see an example of a pathological but common situation Cloudflare sees at their scale that makes other protocols unsuitable for them, therefore they made X choice and this led to Y gains in production (or on dummy workloads, or whatever). As it stands, there's nothing here actually specific to Cloudflare's workload or deployment. It doesn't even state their use-case beyond "small pieces of control plane state (e.g., leadership for replicated databases)"
But it's not a large-scale public deployment yet either. The article says towards the end that they just ran a proof of concept.
Maybe the blog post is just premature. It would be much more valuable if they posted it after actually having run it in production and validated the strengths and weaknesses with real world data.
Agreed on - it took a long time to get to the “so what is new here” vs the broader topic of distributed consensus. Stylistically would prefer more upfront “here is what is novel here”
It feels like somebody prompted an AI agent at CF "why Meerkat is better than Raft" when drafting this blog post.
Everyone cloudflare blog post that gets posted on here shares that quality now
The article lets itself down when it reveals it's had some marketing "polish" added: "where certain resources (like an AI model instance) are stored"
They could have picked something far more concrete (for Cloudflare's business and the type of services they deploy) than "AI model" there, which is clearly picked just to make it seem 'current' and on-trend.