Comment by munksbeer
8 months ago
I'm not a fan or an anti-fan of kafka, but I do wonder about the hate it gets.
We use it for streaming tick data, system events, order events, etc, into kdb. We write to kafka and forget. The messages are persisted, and we don't have to worry if kdb has an issue. Out of band consumers read from the topics and persist to kdb.
In several years of doing this we haven't really had any major issues. It does the job we want. Of course, we use the aws managed service, so that simplifies quite a few things.
I read all the hate comments and wonder what we're missing.
That’s my experience too. I’ve deployed it more than ten times as a consultant and never really understood the reputation for complexity. It “just works.”
I've deployed it a bunch of times and, crucially, maintained it thereafter. It's very complex, especially when troubleshooting pathological behavior or recovering from failures, and I don't see why anyone with significant experience with Kafka could reasonably claim otherwise.
Kafka is perhaps the most aptly named software I've ever used.
That said, it's rock solid and I continue to recommend it for cases where it makes sense.
I know a team that had their Kafka cluster fall over, and they couldn't get it to stay up until eventually their LOB got shut down for being unreliable. I don't know if they were especially bad at their jobs, or their volumes were unreasonable, or what, but it seemed like a bad time for all.
How long is your Kafka down when you cut the cable to Kafka and it needs to fail over?
What happens if the Kafka node fails ?
"the" node? Kafka is a cluster of multiple nodes.
People do lose clusters occasionally.