Comment by supermatt

> partitions in their current form don't

That depends on how you are using partitions. A partition per topic is effectively going to give you exactly that. What you call a key is then just a topic, and hierarchy (including multitenancy and other forms of namespacing) can be implemented via topic naming convention. This isn't even a novel way to use kafka - its quite a common approach in practice.

Obviously this then comes at the cost of throughput - which is exactly why systems that use these approaches are often much slower than partitioned topics in kafka. Even in your object store example there needs to be synchronization across the storage partitions, and that overhead will give you substantially reduced throughput - as you are effectively using a distributed lock for each write to the complete quorum.