Comment by boredandroid
13 years ago
Yeah totally agree. I focused on real-time data because offline data processing is often able to be less principled about the usage of time and doesn't need an explicit log (e.g. many ETL pipelines work this way).
Hi there - I see you're Mr Kafka :-) I wanted to use Kafka instead of Raft for my project, but for my application I couldn't tolerate the (even highly unlikely) possibility of losing a message (when we lose all the nodes in the In-Sync Replica set). I understand why this tradeoff is there (for the clickstream use-case), but I hope it might be possible to retrofit reliable writes to support other use cases as well. The more open-source reliable log services we have the better!
Yup, agreed. Wanna help? :-)
I'd love to help (I opened KAFKA-1050 on the issue), and I even started coding, but I got stuck on how to deal with the rollback problem when a write isn't acknowledged by a quorum. I think it requires a "real" distributed consensus protocol (unlike the current design), which is a fairly big change! It may be that the answer is to use Kafka's efficient storage design with Raft, which is something I explored a little on day 1 of my project: http://blog.justinsb.com/blog/2013/12/07/cloudata-day-1/
First, great article.
Coming across article now was quite serendipitous as I'm in the process of designing a Databus like system myself. In regards to Kafka and Databus, one thing I've been wondering about is why the Databus relay isn't implemented with Kafka? Kafka seems to provide the same semantics (at-least-once in order delivery, clients can pull from arbitrary positions in the stream, etc.). I know Databus provides transaction semantics in the stream and a few other differences, but those differences don't seem too large. Is it because Databus is to be embedded in Espresso? Or maybe even something as simple as two different teams converging on the same solutions?
Anyway, thanks again for the article.