Comment by rashidae
1 day ago
What happens when the DB gets large? How do you handle idempotency? (What if SQS delivers twice?) The cron job is still a single point of failure...
1 day ago
What happens when the DB gets large? How do you handle idempotency? (What if SQS delivers twice?) The cron job is still a single point of failure...
Managing complex scheduled workflows at scale comes with a lot of nuances. This is exactly why we're building DBOS (shameless plug! https://github.com/dbos-inc), which provides durable cron jobs and exactly-once workflow triggering. Since it's just a library on top of Postgres, it doesn't require a centralized scheduler (well, think of Postgres as the coordinator).
One challenge is to guarantee exactly-once processing across software upgrades. DBOS uses the cron-scheduled time as an idempotency key, and tags each workflow execution with a version. We also use the database transactions to guard against conflicting concurrent updates.