← Back to context

Comment by memset

1 year ago

I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq

It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)

6 comments

memset

Reply
GordonS  1 year ago

Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)

  • memset  1 year ago

    Not today. It's a work in progress! There are several iterations that I'm working on:

    1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication

    However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently.

    Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)

justinclift  1 year ago

Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?

  • memset  1 year ago

    It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull.

    The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.