Comment by msteffen

Well, it was just that the models didn't really line up.

To give a somewhat longer description of what Pachyderm does/did:

1. Connect data-scale version control with pipeline orchestrator.

2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.

3. Orchestrator only processes changed shards from one run to the next, instead of all of them.

4. Profit.

That's fine for e.g. one mapreduce, but,

1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)

2. If the overall pipeline changes, you can't re-use results from the first change onward

3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.

So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.

It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!