Comment by redditor98654
7 months ago
I agree on the head of the line blocking problem and that not everyone needs the per partition ordering. For that I have started to use SQS FIFO with the message grouping key being the logical key for the event/resource. This gives me ordering within the key and not extra ordering across keys. So I don’t have the head of line blocking problem.
If I need multiple independent consumers, I just instead publish to SNS FIFO and let my consumers create their own SQS fifo queues that are subscribed to the topic. The ordering is maintained across SNS and SQS. I also get native DLQ support for poison pills and an SQS consumer is dead simple to operate vs a Kafka consumer.
It does not solve all of the mentioned problems like being able to see what the keys are in the queue or lookup by a given key but as a messaging solution that offers ordering for a key, this is hard to beat.
How do you handle DLQ for ordered keys? I assume if you drop a message then you lose semantics
Correct, you lose that ordering but then the only other option is to retry indefinitely. You have to think through what your error handling looks like as is the case with any application.