← Back to context

Comment by stock_toaster

2 hours ago

> The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless.

redis would suffer from the same issue. Possibly even more severely due to being memory constrained?

I'd probably just stuff the "large data" in s3 or something like that, and just include the reference/location of the data in the actual job itself, if it was big enough to cause problems.