Comment by willtemperley
15 days ago
Another thing worth mentioning is it's very similar to the structure of columnar formats like Arrow and Parquet. Anyone with familiarity with these formats could build a decoder in a couple of days. If they don't use FastPFOR.
I really wish they hadn't used FastPFOR. It's a research library and has an incredibly opaque algorithm:
https://ayende.com/blog/199523-C/integer-compression-underst...
FSST is similar in terms of underlying complexity. You need this complexity to get good performance though, it seems from the research.
If there were something better than FastPFOR, we would use it. If something comes up, we can always use a new tag and add it in the future.
There is still a lot to do, for example one can do like-operators on FSST without decompressing it.