Comment by bsaul
5 years ago
Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
5 years ago
Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
I feel as if you're using Kafka and expect guaranteed ordering, then you're using the wrong tool. At best you have guaranteed ordering per partition but then you've tied your ordering/keying strategy to the amount of partitions you've enabled ... which may not ideal.
But, that's speaking from my light experience with it. I'm also curious if there's a better way :-)
Not for Kafka, but we are building Flow [1] to offer deterministic ordering, even across multiple logical and physical partitions, and regardless of whether you're back-filling over history or processing in real-time.
This ends up being required for one of our architectural goals, which is fully repeatable transformations: You must be able to model a transactional decision as a Flow derivation (like "does account X have funds to transfer Y to Z ?", and if you create a _copy_ of that derivation months later, get the exact same result.
Under the hood (and simplifying a bit) Flow always does a streaming shuffled read to map events from partitions to task shards, and each shard maintains a min-heap to process events in their ~wall-time order.
This also avoids the common "Tyranny of Partitioning", where your upstream partitioning parallelism N also locks you into that same task shard parallelism -- a big problem if tasks manage a lot of state. With a read-time shuffle, you can scale them independently.
[1]: https://github.com/estuary/flow
It depends on what the events are, how they are structured.
You get guaranteed ordering at the partition level.
Items are partitioned by key so you also get guaranteed ordering for a key.
If you have guaranteed ordering for a key you can’t get total ordering across all keys but you can get eventual consistency across the keys.
Ultimately if you want ordering you have to design around being eventually consistent.
I don’t read a lot of papers but Leslie Lamports Time, Clocks, and the Ordering of Events in a Distributed System gave me a lot of insight in to the constraints. https://lamport.azurewebsites.net/pubs/time-clocks.pdf
For kafka the default is round robin in each partition. A hash key can let you direct the work to particular partitions. Each partition is guaranteed ordering. Also only one consumer in a consumer group can remove an item from a partition at a time. No two consumers in a consumer group will get the same message.
It's round robin if no key specified otherwise it uses murmur2 hash of the key so the partition for a key is always deterministic.
Just checking the docs it appears the round robin is no longer true after Confluent Platform 5.4. After 5.4 it looks like if no key specified the partition is assigned based on the batch being processed.
> If the key is provided, the partitioner will hash the key with murmur2 algorithm and divide it by the number of partitions. The result is that the same key is always assigned to the same partition. If a key is not provided, behavior is Confluent Platform version-dependent:...
https://docs.confluent.io/platform/current/clients/producer....
2 replies →
At lower data volumes (<10,000 events per minute) it’s perfectly feasible to just use single partition topics and then ordered event delivery is no problem at all. If a consuming service has processing times that means horizontal scaling is necessary then the topic can be repartitioned into a new topic with multiple partitions and the processing application can handle sorting the outputted data to some SLA.
put a timestamp in the message. use a conflict free replicated data type
If you need a guaranteed ordering, timestamps and distributed systems are not friends. See logical / vector clocks.