Comment by reactordev
10 hours ago
Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.
The other system you're using that isn't Postgres can also go down.
Many developers overcomplicate systems. In the pursuit of 100% uptime, if you're not extremely careful, you removed more 9s with complexity than you added with redundancy. And although hyperscalers pride themselves on their uptime (Amazon even achieved three nines last year!) in reality most customers of most businesses are fine if your system is down for ten minutes a month. It's not ideal and you should probably fix that, but it's not catastrophic either.
What I’ve found is that, particularly with internal customers, they’re fine with an hour a month, possibly several, as long as not all of your eggs are in one basket.
The centralization pushes make a situation where if I have a task to do that needs three tools to accomplish, and one of them goes down, they’re all down. So all I can do is go for coffee or an early lunch because I can’t sub in another task into this time slot. They’re all blocked by The System being down, instead of a system being down.
If CI is borked I can work on docs and catch up on emails. If the network is down or NAS is down and everything is on that NAS, then things are dire.
good luck doing anything if kafka is down though
>The other system you're using that isn't Postgres can also go down.
Only if DC gets nuked.
Many developers overcomplicate systems and throw a database at the problem.
Which system is immune to all downtime except the DC getting nuked?
1 reply →
Wow, TIL there was an atomic attack on the capitol in October!
1 reply →
There are a ton of job/queue systems out there that are based on SQL DBs. GoodJob and SupaBase Queues are two examples.
It’s not usable for high scale processing but most applications just need a simple queue with low depth and low complexity. If you’re already managing PSQL and don’t want to add more management to your stack (and managed services aren’t an option), this pattern works just fine. Go back 10-15yrs and it was more common, especially in Ruby shops, as teams willing to adopt Kafka/Cassandra/etc were more rare.
And there are a ton that aren’t.
How so? There are queues that use SQL (or no-SQL) databases as the persistence layer. Your question is more specific to the implementation, not the database as persistence layer itself. And there are ways to address it.
Criticism without a better solution is only so valuable.
How would you do this instead, and why?
Watching a carpenter try to weld is equally only so valuable. I think the explanation is clear.
I prefer using MS Exchange mailboxes for my message queue.
You wouldn't ack the message if you're not up to process it.