It’s not that. It usually ends up in the same storage engine as the main business functionality which leads to IO contention. The some sick and twisted bastard does a join to it from somewhere abhorrent which means it requires some serious heavy lifting to fix it.
This is unfortunately the status quo in the slightly less ideal universe than “seemed like a good idea at the time”
Use SQS or RabbitMQ or something.
Edit: Also if you’re using something SQL for queues you’re going to have to build out your monitoring stack yourself based on your implementation rather than suck up a metrics endpoint with queue stats or pull from cloudwatch.
The benefit of having the queue in the same engine as the main business functionality is that you can do stuff in a single transaction.
Having the queue on its own means you have to handle exceptions like "job is done but queue didn't get updated" or get into distributed transaction which is very different challenge.
Also, operationally having one inherently stateful component (db) versus two (db, queue) makes things a lot simpler to manage, deploy, update, etc.
Not claiming this is a perfect solution, just that the tradeoff isn't obvious.
The point of queues is sequencing, storage and decoupling of events for the sake of scalability and durability.
By putting it in the same transactional store with the same transactional boundaries you’re instantly shooting the whole point.
Not only that, most queues define boundaries between different logical systems where transactional boundaries do exist. At which point your database’s transaction scope is extremely limited in capability.
In the real world of messaging transactions span more than just the database engine.
Contention, architectural separation, you have to build your own monitoring stack, not transactional with concerns outside the database without introducing distributed transactions and risk, no routing or distribution capability, you have to build it yourself, point in time message durability is somewhat dubious depending on your acknowledgement process which of course you had to invent yourself as well.
Not to mention numerous database administrators crying into their cups of coffee.
Enough?
Like I said 20 years of people fucking this up is my experience.
It’s not that. It usually ends up in the same storage engine as the main business functionality which leads to IO contention. The some sick and twisted bastard does a join to it from somewhere abhorrent which means it requires some serious heavy lifting to fix it.
This is unfortunately the status quo in the slightly less ideal universe than “seemed like a good idea at the time”
Use SQS or RabbitMQ or something.
Edit: Also if you’re using something SQL for queues you’re going to have to build out your monitoring stack yourself based on your implementation rather than suck up a metrics endpoint with queue stats or pull from cloudwatch.
The benefit of having the queue in the same engine as the main business functionality is that you can do stuff in a single transaction.
Having the queue on its own means you have to handle exceptions like "job is done but queue didn't get updated" or get into distributed transaction which is very different challenge.
Also, operationally having one inherently stateful component (db) versus two (db, queue) makes things a lot simpler to manage, deploy, update, etc.
Not claiming this is a perfect solution, just that the tradeoff isn't obvious.
The point of queues is sequencing, storage and decoupling of events for the sake of scalability and durability.
By putting it in the same transactional store with the same transactional boundaries you’re instantly shooting the whole point.
Not only that, most queues define boundaries between different logical systems where transactional boundaries do exist. At which point your database’s transaction scope is extremely limited in capability.
In the real world of messaging transactions span more than just the database engine.
It’s just bad architecture. Full stop.
2 replies →
So use a separate database.
Is contention with the application your only objection? That’s pretty weak.
What’s the real compelling reason not to use a database? You haven’t said.
Contention, architectural separation, you have to build your own monitoring stack, not transactional with concerns outside the database without introducing distributed transactions and risk, no routing or distribution capability, you have to build it yourself, point in time message durability is somewhat dubious depending on your acknowledgement process which of course you had to invent yourself as well.
Not to mention numerous database administrators crying into their cups of coffee.
Enough?
Like I said 20 years of people fucking this up is my experience.
9 replies →
I dunno. If it’s usually the same problem, it should be easy to identify and fix.
Unless we’re talking about a high load, there should be no problem doing this.
It’s only the same problem until someone dumps 25,000 entries in your queue.
4 replies →