Comment by brasetvik
3 years ago
It'd be great to get a fairly standard way of doing this. :)
Having worked in this problem space a bit recently, I find this part a bit too optimistic:
> The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.
The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer.
Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though.
Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences.
The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup)
A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.
Perhaps the time problem can be handled with an eventually consistent Lamport clock system
Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes.
Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.
> Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think.
My point was that even if you have a single non-scaleout database with a single time source, a sequence or a timestamp or a combination of both isn't as reliable a sequencer as you might think, unless you have at most one writer.
Thus, I think a "standard" should encourage a cursor concept that can use something that may reliably provide _all_ changes. If you have a single writer, you have a pretty easy job implementing that, as a plain sequence would work. (A timestamp could still break on clock adjustments, though)
This pertains to the "data replication" part of the listed goals, where getting everything is more important than in e.g. a social media news feed style thing where chronological order may be tenable - or less consequential if an item is missed.
2 replies →