Show HN: Waymark – A workflow engine that compiles jobs instead of replaying
2 days ago (github.com)
Hey everyone! @mozgiii and I have been working on Waymark for the better part of the last year. Waymark’s an open source durable execution library with similar functionality to Temporal or DBOS.
I fell in love with durable execution a few years ago, after working at a company with a heavy pubsub architecture. In comparison to job orchestrators that force a certain task design or special syntax, many of these durable libs lets you write business logic in regular code: `asyncio.gather` to parallelize tasks, `for` loops, conditionals, and let the runtime figure out how to run them efficiently across a cluster of nodes. But most of the current solutions require a heavy broker, runtime hacks to try to ensure deterministic workflow logic, and feel like you have to learn a whole new ecosystem just to get started.
A few unique design decisions in waymark:
— Write jobs in normal code (right now Python but JS coming soon). Instead of running control flow as-is, which requires booting up an interpreter to replay past state, we’ll just compile your code directly. We walk your AST and map your logic to our stackless VM bytecode. We support variables, conditionals, await, sleep, helper functions, loops... basically what you’d expect in workflow logic.
— Since we’re compiling a language subset, we can proactively flag errors in your workflows locally instead of when they run in production.
— It only requires Postgres as the task broker. We’ve spent a lot of time tuning our design to play nicely with its strengths and limitations; our current benchmarks show around 1500 workflows/second but we still have some more optimization work to do here.
— Actions plug-and-play with FastAPI’s dependency injection syntax. So if you already have an ecosystem based on typehinting/DI, it should fit like a glove.
It allows for parallelization of workflows across one machine or a whole cluster. The bulk of the work per node is done by our central Rust core - which distributes workflows to properly saturate all cores. The idea is to be as batteries included as possible with the broker, webapp, and runtime.
What to use it for? It works great as the runner loop controlling agents, but it also works well for basically any task that you need to kick off in the background. It’s largely replaced our way of writing api code. Most frontend actions at this point just push to a simple REST endpoint that kick off a background job.
I spent an embarrassing amount of time making our https://waymark.sh site interactive - it’s probably the best place to learn more. We’re starting with Python support and looking for community input on what language to ship next. Would love to hear your thoughts!
looks a lot like temporal
Nice project! (And website!)
Great to see some experimentation in this space. I really like the idea of 'make this async function a background process' using a simple annotation.
What kind of IR do you use? I often wonder what a WASM based durable execution engine would look like. And what that pattern would unlock.
This is so cool. I spent years at Pachyderm (YC...something) banging our head against:
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
Cool thought! We have an internal version of map reduce implemented in waymark that worked pretty well (though uses Redis for its intermediate values so it's technically not an end-to-end integrated solution).
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Where did you hit the most snags with caching?
Well, it was just that the models didn't really line up.
To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!
Yeah, that's an interesting use case. The problem for you with a compiler would be how to get it to work with all of the efficient python modules like pandas, script, numpy, pytorch etc that all implement their backend in C with intrinsics/GPU usage.
Durable execution has been on my radar for a minute but, although I've had a few pubsub usecases here and there, I've never had much call for long-running jobs that weren't simply cheaper to restart, AFAIR.
With agentic workloads, though—especially trending towards high network calls and smaller models—I do find a lot of concurrency to be a pain in the ass to pick up where we left off. Is that what "durability" ameliorates?
I just wonder what I'm leaving on the table.
Yep, that’s exactly it. imo the main value add for agents:
- Each LLM call and tool call becomes a checkpoint. Without durable execution you're pretty much stuck doing db saves/restores if you detect that this agent might have a previous context window. Agent frameworks are fundamentally just harnesses around the underlying model: you `while loop` until some server gives you some payload back, you add that to your context window list, and you keep going until it terminates. Durable execution moves that loop into the workflow body itself (a quick example here https://www.waymark.sh/python/tutorials/building-a-from-scra...).
- Long waits. Sometimes you do want your agent to wait for an arbitrary amount of time (ie. wait for this compilation job to finish - look again in 5mins, or maybe 5 hours). Durable execution makes sure that sleep timer will wake up regardless of what happens in the system state.
- Easier monitorability. Instead of having to sift through interleaved logs to figure out what was going on, you can see every large function call with a clean input/output contract. Useful especially in tracing down raised Exceptions and the data flow that it took to get there. Good for agents and regular code alike.
Of course, if your system is largely idempotent or your agents are mostly classifiers that are working in short internals - a simple db save/restore can work perfectly well too.
Durable execution and orchestration are essential in some specialised uses, such as regulatory reporting for financial institutions, or other snapshot based data pipelines. The transition is happening, and temporal and conductor are the usual tools there, but good tools are not always behind good solutions to put it lightly.
I like this one, as in the context of workloads like these, going with anything more complex than a Postgres is overkill.
I completely understand that it might be because I've never suffered the pain associated with the thing you're purporting to solve, but I really don't understand what Waymark does.
I find this really common on HN - someone posts a project that addresses a problem and all the intro and docs assume you understand the problem and couch the discussion in somewhat domain specific language.
It might be interesting to me. It might even be relevant to me and I don't know it yet. The problem is I need to work really hard to get to the point of understanding that.
This is an appeal to authors of such posts: please give a motivation that provides the necessary background to understand the problem, and ideally do so in a way that a technical but non specialist reader can understand.
How many tasks per workflow for the 1500/sec benchmark?
The idea makes sense and the work is great. Some feedback I have is that it becomes hard to try the project out when postgres is a mandatory dependency: I understand that there are scalability reasons why that choice was made, but it would be neat if sqlite was an option.
We'll look into it! The db layer is already modeled basically like a plugin; should be pretty easy to add in whatever database support you want. Sign up to the newsletter on the website if you want the updates when it's ready :)
so Temporal without the replay, Step Functions without the JSON -pretty slick
Any plans for Java?
Dotnet CIL + sqlite please?
This looks like something that will break easily with a newer version of Python or even with the use of third-party packages. It seems to couple too tightly with the user's application code.
It is too async heavy as per its readme -- this is very off-putting. Not everyone cares for async. There is no usage shown with sync code.
MIT licensed, just clarified in the repo.
As for async - background jobs are better modeled as async because of their natural parallelism across nodes imo. Logic in actions themselves are free to be as synchronous as you want. Why the preference?
I dispute the "better" assertion. Anyhow, I don't know how familiar or not you are with sync code, but it fully supports background jobs using queues with threads/interpreters/processes. It has been around for decades, far longer than async.
Also, it is okay to run something in the foreground. Not everything needs to run in the background. Sometimes it's better to wait and have assurance of completion.
There is no natural parallelism of async code in Python. The coroutines run in a single thread.
I don’t understand these kinds of strong emotions async python triggers in some people. It reminds me of the irrational hate towards systemd in the Linux world.
This is a platform for running workflows distributed on multiple machines, sometimes long-running and with lots of IO. Every single contender in the space advertises the same code, from temporal to hatchet to dagster. Because it makes sense to write it this way.
Huh. I hate neither async nor systemd. There is nothing wrong with async, but if my code is sync, what am I to do?
Also, the claim made that something is "better modeled as async" is wholly unqualified.
1 reply →
[dead]
[flagged]
How do you handle code changes for workflows already in flight? Say an instance is sleeping inside a loop for three days and someone deploys a new version of that loop. Is the compiled bytecode pinned per instance, or do you map the old position onto the new code?
[flagged]
[flagged]