Comment by _pdp_

For reasons I recently got acquainted with enterprise scheduling solutions. IBM/HCL workload automation specifically - the company is actively looking for a replacement but nothing except for the chinese sponsored open source comes close it seems to me (and that is a step too far for management...).

The thing you miss with scripting is that a good workflow engine essentially provides a simple and well-defined layer allowing you to couple many concurrent processes and providing ressource management. Curiously out of the box all the opensource workflow tools do so too - like you said people (Airflow has pools, but thrn ...) essentially built very weird DSLs to do high-level composition and there you can indeed rather easily code it yourself. I assume the companies originating these have integrations for the other bits too while everyone else does fake elastic scaling with some halfbaked workarounds. And from time to time someone comes up with a new DSL and some GUI - ignoring all the hard parts.

Main limitation of these kinds of products is that the graph/workflow itself is not very expressive of dataflow options. Eventually you need map, reduce, and state machines and likewise concurrency and durable execution. It's not easy to have a high level declarative tool but also expose powerful programming primitives; at some point technical need-to-knows cross the point where you might as well have the user write code.

I think the last point is key. These tools are used by those who can’t program, too scared by the learning barrier, or trapped in a SaaS/Security ecosystem that bans user scripts. All three of these can be solved by an LLM putting together scripts in a sandbox environment.