Comment by cklee
1 year ago
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban
Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while maintaining compatibility.
Alternatively, we could consider developing a core job queue library in Rust, with language-specific bindings. This would provide a robust, cross-language solution while leveraging the performance and safety benefits of Rust.
If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq.
Also, pgmq can run as a TLE (trusted language extension), so you can install it into cloud hosted Postgres solutions like Supabase. We're using pgmq, and it's solid so far.
What I like about https://github.com/tembo-io/pgmq is that it can be used with any programming language and does not require any background workers or running a binary in addition to postgres.
I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq
It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)
Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)
Not today. It's a work in progress! There are several iterations that I'm working on:
1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication
However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently.
Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)
Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull.
The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.
2 replies →
This would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.
River ( https://riverqueue.com ) is a Postgres background job engine written in Go, which also has insert only clients in other languages. Currently we have these for Ruby and Python:
https://github.com/riverqueue/riverqueue-ruby
https://github.com/riverqueue/riverqueue-python
This is the exact use case I'm running into right now. I've been looking at BullMQ some it has good typescript support, and is working towards a 1.0 for python. But, I have tried it out in a production stack yet
We have been using bullmq in production for just over a year. It is a piece of technology that our team doesn't have to think about, which is pretty much all I could ask for.
We did end up adding some additional generics which allows us to get strong typing between producers and consumers. That I think has been a key piece of making it easy to use and avoiding dumb mistakes.
Qless "solves" this problem (in redis) by having all core logic written as lua and executed in redis.
You could take a similar approach for pg: define a series of procedures that provide all the required functionality, and then language bindings are all just thin wrappers (to handle language native stuff) around calls to execute a given procedure with the correct arguments.
River is my go to in Golang, it’s really handy to have transactional queuing with a nice little ui.
A common schema is one nice thing, but imho the win of these db backed queues is being able to do things, including enqueue background jobs in a single transaction. e.g. create user, enqueue welcome email - both get done, or not - with redid-based, this is ... not usually a thing; if you fail to do one, it's left half done, leading to more code etc
p.s. I maintain a ruby equivalent called QueueClassic
For JavaScript and Perl there is already Minion, which relies on listen/notify + FOR UPDATE SKIP LOCKED.
https://github.com/mojolicious/minion.js
https://github.com/mojolicious/minion