Yep. But we all know that one machine can and will fail (or be patched and restarted), so the log needs to be distributed.
Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.
In all seriousness, I’d take a “s3 is all you need for durable workflows” and use it in data processing applications that move data from s3 -> s3 with no other dependencies.
Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows.
A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?
I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.
Yep. But we all know that one machine can and will fail (or be patched and restarted), so the log needs to be distributed.
Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.
Not a bad way to do things.
Wait no further. It's already happening.
One reason why a "logs are all you need" solution may fail: untrusted-log-as-injection[1].
Check those SBOM, and don't forget to include their CICD pipelines[2].
[1] https://github.com/jqwik-team/jqwik/issues/708#issuecomment-...
In all seriousness, I’d take a “s3 is all you need for durable workflows” and use it in data processing applications that move data from s3 -> s3 with no other dependencies.
Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?
Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.
I swear it didn't occur to me that that mean WAL, makes much more sense now LOL
[dead]
I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows.
A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.
Is it common to use logs as a proxy for write-ahead logs?
1 reply →
I read the parents comment as sarcasm and not a serious suggestion.
Log as in the structure.
Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?
I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.
Shortly followed by:
"Sockets are all you need for durable workflows" and then finally "Kernel primitives are all you need for durable workflows."
But seriously, part of being a professional is using the right tool for the job.