Comment by gopalv

> does duckdb support querying custom storage formats?

No, this is still parquet being loaded in.

The "trick" is running through sort order combinations which result in maximum packing in a Dremel style data format (like Parquet).

Here's the trick explained in enough detail to understand, but not enough to copy as-is

https://cloud.google.com/blog/products/bigquery/inside-capac...

Search for the section with

"Capacitor builds an approximation model that takes into account all relevant factors and comes up with a reasonable solution."

or a more detailed dive in the paper (by the favourite author of performance engineers, Daniel Lemire)

https://arxiv.org/abs/1207.2189