Comment by hparadiz
1 year ago
I've done this with mysql. Never do one at a time if you have jobs per minute over 30. It won't scale. Instead have the job dispatcher reserve 100 at a time and then fire that off to a subprocess which will subsequently fire off a process for each job. A three layer approach makes it much easier to build out multiserver. Or if you don't want the headache just use SQS which is pretty much free under 1 million jobs.
Yeah it's very basic and limited.
However if I am about to use DB as a job queue for budget reasons, I'd make sure the job doesn't get too complicated.
For me it was a lot of small jobs.
I was able to get it up to 3500 jobs an hour and likely could have gone far past that but the load on the MySQL server was not reasonable
I was able to push PqQueuer to 25k jobs a second in my benchmarking script.