Comment by jorangreef
1 year ago
I’m not disagreeing that diskless crash recovery protocols exist.
In fact, an early version of TigerBeetle implemented one of these from Cowling and Liskov’s VSR’12 paper.
However, since then, we invested in TigerBeetle’s stable storage, for reasons which I won’t go into further here.
If you are curious to learn more about all the stable storage techniques in TigerBeetle in particular (again, this is not trying to suggest that VSR can’t also run without stable storage, or to deny other techniques such as object storage or tiering!), but if these things are interesting for you, and if you do want to learn more, then I’d recommend diving into “Durability and the Art of Consensus”.
I've read that paper, and all of the papers that TigerBeetle mentions in any and all of its docs/blog posts/etc. Stable storage doesn't matter to the points I'm trying to shine a light on here. It's not "diskless" that I'm talking about, in fact I'm not talking about "crash recovery" at all -- !
> I’ve read that paper
To be clear, “Durability and the Art of Consensus” is not a paper.
> tl;dr: nodes don't necessarily persist state to local disk
In the consensus literature, this is sometimes referred to as “diskless crash recovery”. For example, see work by Dan Ports.
> Local recovery actions, or any other kinds of node-specific details, have no relevance or influence on the information communicated thru the the global consensus protocol.
This goes directly against the central finding of PAR, which gives counter-examples where your statement does not universally hold true.
Again, it’s not intuitive. And that’s why it won FAST18—because it says “everything we know is wrong”. Complete red pill and mindbender.
PAR isn't any kind of panacea or golden rule for nodes in a distributed system, it describes properties of nodes that meet very narrowly-defined requirements, which are in no way universal, and which are in no way requirements for those nodes to participate in the distributed system.
More broadly, there's no concept of "crash recovery" at the system level, which has any meaningful utility. Nodes are either there or they're not there, exactly how they crash or recover from their crashes are irrelevant to the overall distributed system, insofar as if a crashed-and-recovered node comes back online, it's gonna need to re-sync with its peers before it can talk to anyone else, and that's not anything to do with "crash recovery" related to local disk or anything like that (which is all implementation details of the node itself) -- it's just normal node synchronization, orthogonal to any state storage stuff of the node.
PAR is something that your system can maybe implement, it's not any kind of rule or definition or requirement that all systems of some classification must satisfy..!
1 reply →