Comment by BurningFrog
9 years ago
In my experience, the cost of adding and maintaining another storage subsystem to a project is often hugely underestimated. It's easy to see the benefits and ignore the costs.
If I can solve a problem reasonably well by adding a table to my Postgres DB, that will always beat out adding the specialized ScrewdriverDB that does it perfectly.
I agree. If it sounds simple, then you are probably not thinking hard enough.
Think for example how to do backups. If you have a database and completely separate queue system your application state is distributed into two places. This means taking consistent backups is not straightforward. You can of course try to work around this on the application level (database as single source of truth, queues as "nice to have" -thing, but this makes things more complicated for the app).
> This means taking consistent backups is not straightforward.
And they're not even straightforward to begin with...
Depends on what you're adding. Running Redis is dead simple and easy to have visibility into. RabbitMQ / Kafka are much larger undertakings.
If you already had a need for a durable database, and so you properly implemented Postgres streaming-archiving + disaster recovery, at much personal effort for your tiny startup... and you now need a durable queue as well... then "just installing Redis" won't get you there. You'll need to that whole ops setup all over again for Redis. Whereas, if your queue is in Postgres, the ops problem is already solved.
If you have a full-time ops staff, throwing another marginal piece of infrastructure on the pile isn't much of an issue. If you're your own "ops staff of one", each moving part of your infrastructure—that you need to ensure works, to guarantee you won't lose client data—is another thing slowing down your iteration speed.
You're making a lot of assumptions that aren't terribly valid for most use cases. Not to say that strict consistency requirements don't exist, they certainly do, but they are the exception rather than the norm. A one person startup doesn't usually care too much about losing a bit of client data.
> If you have a full-time ops staff, throwing another marginal piece of infrastructure on the pile isn't much of an issue.
That's a rather insulting description of Redis, care to elaborate?
3 replies →
Depends on the scale mostly. People often forget how not every project is Google/Facebook/Twitter.
RabbitMQ is on docker, its more or less the same work to launch as Redis these days.
(It can take a bit more tuning so I think it is unfair to say it is the SAME work, but it is seriously not a huge deal to run RabbitMQ in the post docker world)
Is Docker really necessary for this? I've never had any trouble just installing RabbitMQ from the package manager, and running it with only a tiny bit of initial configuration.
1 reply →