← Back to context

Comment by teagee

12 hours ago

Any code or blog written by Adam is worth spending some time on.

It will be interesting to see how the tasks framework develops and expands. I am sad to see the great Django-Q2 lumped in with the awful Celery though.

20 comments

teagee

Reply
sgt  33 minutes ago

I've been using Celery for years. What is the major issues you have with it and how does Django Q2 help?

I also use Kafka on other tech stacks but that's another level completely and use case.

boxed  21 minutes ago

I tried django-q and I thought it was pretty terrible. The worst was that I couldn't get it to stop retrying stuff that was broken. Sometimes you ship code that does something unexpected, and being able to stop something fast is critical imo.

Fundamentally I think the entire idea behind celery and django-q is mostly misguided. People normally actually need a good scheduler and a bring-your-own queue in tables that you poll. I wrote Urd to cover my use cases and it's been rock solid.

blorenz  9 hours ago

I’m currently stuck with the tech debt of Celery myself. I understand that! Does Django Tasks support async functions?

hintoftime  10 hours ago

Why is celery awful?

  • JimDabell  6 hours ago

    > The Many Problems with Celery:

    — https://steve.dignam.xyz/2023/05/20/many-problems-with-celer...

    > The problems with (Python’s) Celery:

    — https://docs.hatchet.run/blog/problems-with-celery

    > Dramatiq motivation:

    — https://dramatiq.io/motivation.html

    Here are some alternatives:

    Dramatiq: https://github.com/Bogdanp/dramatiq

    RQ: https://github.com/rq/rq

    Huey: https://github.com/coleifer/huey

    Hatchet: https://github.com/hatchet-dev/hatchet

  • leobuskin  8 hours ago

    It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).

  • tclancy  9 hours ago

    Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times.

    And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.

    https://flower.readthedocs.io/en/latest/

  • akoumjian  8 hours ago

    Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:

    - your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes

    There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.

    Honestly, it's a great education.

    • ffsm8  6 hours ago

      > your side effects (e.g. database writes) aren't idempotent

      What does idempotent mean in this context, or did you mean atomic/rollback on error?

      I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips

      1 reply →

jonatron  11 hours ago

I'm of the opinion that django task apps should only support a single backend. For example, django-rq for redis only. There's too many differences in backends to make a good app that can handle multiple. That said, I've only used celery in production before, and I'm willing to change my mind.