Comment by maeln

5 months ago

So, as I understand, you describe the structure of your data in an SDL and then the compressor can plan a strategy on how to best compress the various part of the data ?

Honestly looks incredible. Could be amazing to provide a general framework for compressing custom format.

3 comments

maeln

Reply
terrelln  5 months ago

Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in C++ or Python to parse your format. And this code is compression side only, so the decompressor is agnostic to your format.

[0] https://openzl.org/api/c/graphs/sddl/

  • maeln  5 months ago

    Now I cannot stop thinking about how I can fit this somewhere in my work hehe. ZStandard already blew me away when it was released, and this is just another crazy work. And being able to access this kind of state-of-the-art algo' for free and open-source is the oh so sweet cherry on top

  • touisteur  5 months ago

    How happy I am to have all written/read data going through a DSL. On to generating the code to make OpenZL happy...