Comment by OutOfHere
2 days ago
I am not convinced that using a special software for "durable workflows" is necessary. If one has a stateful message queue or job task queue, e.g. RabbitMQ or Celery, one can use it. Irrespective, many jobs can be made idempotent. The most that you ought to residually need is a column in an existing table of your own database which keeps track of what remains to be done.
Given the above, it would seem that durable workflow software is pushed forward by those who have a surplus of VC money to spend. As for the vendors, there is no shortage of people trying to sell you things that you don't need.
I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.
That argument comes down to the scalability of RabbitMQ or one's database, both of which can scale fairly well, but require tuning. In the absolute worst case, one would have to use a distributed cloud database, e.g. AWS Aurora or AWS DynamoDB, otherwise a self-hosted one, e.g. TiDb or YugabyteDb, but far less than 1% of users would even need anything like it.
In the pre-AI era, the argument of using a third party tool or service even had some weight, but today, AI can even do much of the heavy work when pointed in the right direction wrt using the aforementioned. For the majority of users, a SQLite database will do the job.