Comment by supermatt
8 months ago
> "Do away with partitions"
> "Key-level streams (... of events)"
When you are leaning on the storage backend for physical partitioning (as per the cloud example, where they would literally partition based on keys), doesnt this effectively just boil down to renaming partitions to keys, and keys to events?
That's one way to look at this, yes. The difference being that keys actually have a meaning to clients (as providers of ordering and also a failure domain), whereas partitions in their current form don't.
> partitions in their current form don't
That depends on how you are using partitions. A partition per topic is effectively going to give you exactly that. What you call a key is then just a topic, and hierarchy (including multitenancy and other forms of namespacing) can be implemented via topic naming convention. This isn't even a novel way to use kafka - its quite a common approach in practice.
Obviously this then comes at the cost of throughput - which is exactly why systems that use these approaches are often much slower than partitioned topics in kafka. Even in your object store example there needs to be synchronization across the storage partitions, and that overhead will give you substantially reduced throughput - as you are effectively using a distributed lock for each write to the complete quorum.