Comment by akoumjian
10 hours ago
Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:
- your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes
There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.
Honestly, it's a great education.
> your side effects (e.g. database writes) aren't idempotent
What does idempotent mean in this context, or did you mean atomic/rollback on error?
I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
In the context of background jobs idempotent means that if your job gets run for a second time (and it will get run for a second time at some point, they all do at-least-once delivery) there aren't any unfortunate side effects to that. Often that's just a case of checking if the relevant database updates have already been done, maybe not firing a push notification in cases of a repeated job.
In my experience async job idempotency is implemented as upserts. Insert all job outputs on the first run. Do (mostly) nothing on subsequent runs. Maybe increment a counter or timestamp.
From your experience, what is a better alternative guys?
There’s no alternative (while prototyping), and anything else is better (when you properly defined your case).
DjangoQ2 is a fine alternative during early development
Not the comment that you replied to but I use my own Urd. It's a fancier Cron that you can stop fast. Which is imo what you normally want.
Task queues are like email. It's what everyone is used to so people ask for more of it, but it's not actually good/the right tool.