Comment by sorentwo
1 day ago
Transactions around fetching/updating aren't trivial, that's true. However, the work that you're doing _is_ regular activity because it's part of your application logic. That's data about the state of your overall system and it is extremely helpful for it to stay with the app (not to mention how nice it makes testing).
Regarding overall throughput, we've written about running one million jobs a minute [1] on a single queue, and there are numerous companies running hundreds of millions of jobs a day with oban/postgres.
[1]: https://oban.pro/articles/one-million-jobs-a-minute-with-oba...
Appreciate the response, I'm learning some new things about the modern listening mechanisms for DBs which unlock more than I believed was possible.
For your first point - I would counter that a lot of data about my systems lives outside of the primary database. There is however an argument for adding a dependency, and for testing complexities. These are by and large solved problems at the scale I work with (not huge, not tiny).
I think both approaches work and I honestly just appreciate you guys holding Celery to task ;)