← Back to context

Comment by colinchartier

5 years ago

Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!

35 comments

colinchartier

Reply
macksd  5 years ago

I think the point of interest was 32 cores to handle what sounds like 10 messages per second at most. That's not really a ton of throughput... It's certainly a valid point that an awful lot of uses cases don't need Twitter-scale firehoses or Google-size Hadoop clusters.

  • colinchartier  5 years ago

    Ah, the database does a lot more than just pub/sub - especially since the high traffic pub/sub goes through redis. I guess my point was that we never regretted setting up postgres as the "default job queue" and it never required much engineering work to maintain.

    For an example, it handles stripe webhooks when users change their pricing tier - if you drop that message, users would be paying for something they wouldn't receive.

  • rowanG077  5 years ago

    It said nothing about the distribution of traffic. It might well be thousands and thousands of pub sub messages at some point of the day and 0 for others.

rockwotj  5 years ago

Fwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...

threeseed  5 years ago

But Kafka does significantly more.

And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.

  • speed_spread  5 years ago

    Limit the types of server used to reduce system complexity. If you can have all your business state in the same place, ops are much easier.

    Kafka does more for streaming data, but doesn't do squat for relational data. You always need a database, but you sometimes can get by without a queuing system.

pts_  5 years ago

That's the job of a DevOps engineer not a backend developer attempted to be overworked.