Comment by kukkeliskuu
2 days ago
I think we should stop calling these systems eventually consistent. They are actually never consistent. If the system is complex enough and there are always incoming changes, there is never a point in time in these "eventually consistent systems" that they are in consistent state. The problem of inconsistency is pushed to the users of the data.
Someone else stated this implicitly, but with your reasoning no complex system is ever consistent with ongoing changes. From the perspective of one of many concurrent writers outside of the database there’s no consistency they observe. Within the database there could be pending writes in flight that haven’t been persisted yet.
That’s why these consistency models are defined from the perspective of “if you did no more writes after write X, what happens”.
They are consistent (the C in ACID) for a particular transaction ID / timestamp. You are operating on a consistent snapshot. You can also view consistent states across time if you are archiving log.
"... with your reasoning no complex system is ever consistent with ongoing changes. From the perspective of one of many concurrent writers outside of the database there’s no consistency they observe."
That was kind of my point. We should stop callings such systems consistent.
It is possible, however, to build a complex system, even with "event sourcing", that has consistency guarantees.
Of course your comment has the key term "outside of the database". You will need to either use a database or built a homegrown system that has similar features as databases do.
One way is to pipe everything through a database that enforces the consistency. I have actually built such an event sourcing platform.
Second way is to have a reconciliation process that guarantees consistency at certain point of time. For example, bank payments systems use reconciliation to achieve end-of-day consistency. Even those are not really "guaranteed" to be consistent, just that inconsistencies are sufficiently improbable, so that they can be handled manually and with agreed on timeouts.
The way you're defining "eventually consistent" seems to imply it means "the current state of the system is eventually consistent," which is not what I think that means. Rather, it means "for any given previous state of the system, the current state will eventually reflect that."
"Eventually consistent," as I understand it, always implies a lag, whereas the way you're using it seems to imply that at some point there is no lag.
I was not trying to "define" eventually consistent, but to point out that people typically use the term quite loosely, for example when referring to the state of the system-of-systems of multiple microservices or event sourcing.
Those are never guaranteed to be in consistent state in the sense of C in ACID, which means it becomes the responsibility of the systems that use the data to handle the consistency. I see this often ignored, causing user interfaces to be flaky.
Inconsistent sounds so bad :)
1 reply →
They eventually become consistent from the frame of a single write. They would become consistent if you stopped writes, so they will eventually get there
Both of your statements are true.
But in practice we are rarely interested in single writes when we talk about consistency, but the consistency of multiple writes ("transactions") to multiple systems such as microservices.
It is difficult to guarantee consistency by stopping writes, because whatever enforces the stopping typically does not know at what point all the writes that belong together have been made.
If you "stop the writes" for sufficiently long, the probability of inconsistencies becomes low, but it is still not guaranteed to be non-existant.
For instance in bank payment systems, end-of-day consistency is handled by a secondary process called "reconciliation" which makes the end-of-day conflicts so improbable that any conflict is handled by a manual tertiary process. And then there are agreed timeouts for multi-bank transactions etc. so that the payments ultimately end up in consistent state.
That’s a fair point. To be fair to the academic definitions, “eventually consistent” is a quiescent state in most definitions, and there are more specific ones (like “bounded staleness”, or “monotonic prefix”) that are meaningful to clients of the system.
But I agree with you in general - the dynamic nature of systems means, in my mind, that you need to use client-side guarantees, rather than state guarantees, to reason about this stuff in general. State guarantees are nicer to prove and work with formally (see Adya, for example) while client side guarantees are trickier and feel less fulfilling formally (see Crooks et al “Seeing is Believing”, or Herlihy and Wing).
I have no beef with the academic, careful definitions, although I dislike the practice where academics redefine colloquial terms more formally. That actually causes more, not less confusion. I was talking about the colloquial use of the term.
If I search for "eventual consistency", the AI tells me that one of the cons for using eventual consistency is: "Temporary inconsistencies: Clients may read stale or out-of-date data until synchronization is complete."
I see time and time again in actual companies that have "modern" business systems based on microservices that developers can state the same idea but have never actually paused to think that you something is needed to do the "synchronization". Then they build web UIs that just ignore the fact, causing application to become flaky.
> They are actually never consistent
I don't see it this way. Let's take a simple example - banks. Your employer sends you the salary from another bank. The transfer is (I'd say) eventually consistent - at some point, you WILL get the money. So how it can be "never consistent"?
Your example is too simple to show the problem with the "eventual consistency" as people use the term in real life.
Let's say you have two systems, one containing customers (A) and other containing contracts for the customers (B).
Now you create a new contract by first creating the customer in system A and then the contract on system B.
It may happen that web UI shows the contract in system B, which refers to the customer by id (in system A), but that customer becomes visible slightly after in system A.
The web UI has to either be built to manage the situation where fetching customer by id may temporarily fail -- or accept the risk that such cases are rare and you just throw an error.
If a system would be actually "eventually consistent" in the sense you use the term, it would be possible for the web UI to get guarantee from the system-of-systems to fetch objects in a way that they would see either both the contract and the customer info or none.
Because I will have spent it before it becomes available :)
For the record (IMO) banks are an EXCELLENT example of eventually consistent systems.
They're also EXCELLENT for demonstrating Event Sourcing (Bank statements, which are really projections of the banks internal Event log, but enough people have encountered them in such a way that that most people understand them)
I have worked with core systems in several financial institutions, as well as built several event sourcing production systems used as the core platform. One of these event sourcing systems was actually providing real consistency guarantees.
Based on my experience, I would recommend against using bank systems as an example of event sourcing, because they are actually much more complex than what people typically mean when they talk about event sourcing systems.
Bank systems cannot use normal event sourcing exactly because of the problem I describe. They have various other processes to have sufficiently probable consistency (needed by the bank statements for example), such as "reconciliation".
Even those do not actually "guarantee" anything, but you need tertiary manual process to fix any inconsistencies (on some days after the transaction). They also have timeouts agreed between banks to eventually resolve any inconsistencies related to cross-bank payments over several weeks.
In practice this means that the bank statements for source account and target account may actually be inconsistent with each other, although these are so rare that most people never encounter them.
If the bank transaction is eventually consistent, it means that the state can flip and the person receiving will "never" be sure. A state that the transaction will be finished later is a consistent state.
Changing terminology is hard once a name sticks. But yeah, "eventual propagation" is probably more accurate. I do get the impression that "eventual consistency" often just means "does not have a well-defined consistency model".
Yes, I agree. I don't really believe we can change the terminology. But maybe we can get some people to at least think about the consistency model when using the term.
> If the system is complex enough and there are always incoming changes, there is never a point in time in these "eventually consistent systems" that they are in consistent state.
Sure, but that fact is almost never relevant. (And most systems do have periods where there aren't incoming changes, even if it's only when there's a big Internet outage). What would be the benefit in using that as a name?
It is highly relevant in many contexts. I see in my work all the time that developers building frontends on top of microservices believe that because the system is called "eventually consistent", they can ignore consistency issues in refences between objects, causing flaky apps.
> I see in my work all the time that developers building frontends on top of microservices believe that because the system is called "eventually consistent", they can ignore consistency issues in refences between objects, causing flaky apps.
If someone misunderstands that badly, why would one believe they would do any better if it was called "never consistent"?
1 reply →
Just like Git. Why bother with all these branches, commits and merges?
Just make it so everyone's revision steps forward in perfect lockstep.
Branches, commits and merges are the means how people manually resolve conflicts so that a single repository can be used to see a state where revision steps forward in perfect lockstep.
In many branching strategies this consistent state is called "main". There are alternative branching stragies as well. For example the consistent state could be a release branch.
Obviously that does not guarantee ordering across repos, hence the popularity of "monorepo".
Different situations require different solutions.
> If the system is complex enough and there are always incoming changes
You literally don't understand the definition of eventual consistency. The weakest form of eventual consistency, quiescent consistency, requires [0]:
Emphasis on the "updates stop[ping] at some point," or there being only "finitely many updates." By positing that there are always incoming changes you already fail to satisfy the hypothesis of the definition.
In this model all other forms of eventual consistency exhibit at least this property of quiescent consistency (and possibly more).
[0] https://www.microsoft.com/en-us/research/wp-content/uploads/...
My point was kind of tongue-in-cheek. Like the other comment suggests, I was talking about how people actually use the term "eventually consistent" for example to refer to system-of-systems of multiple microservices or event sourcing systems. It is possible to define and use the terms more exactly like you suggest. I have no problem with that kind of use. But even if you use the terms more carefully, most people do not, meaning that when you talk about these systems using the misunderstood terms, people may misunderstand you although you are careful.
The GP proposed that the definition should be changed. That in no way implies a lack of understanding of the present definition.