Comment by simonw
7 hours ago
"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.
I expect that if your overall data is less than a GB this trick will work really well for you.
7 hours ago
"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.
I expect that if your overall data is less than a GB this trick will work really well for you.
Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation.
The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without. This is from the same author as the parquet library mentioned in the OP https://github.com/hyparam/icebird
I think it depends on (1) customer expectations for freshness and (2) scale (both for the cubes and for the customer data in toto). There are many types of customer facing dashboards where giving "live data" is a bad idea for them and for you. And recomputation is indeed a problem, but if the volume of data isn't that high to begin with, it's probably easier than setting up an incremental pipeline architecture vs. a grouping set query in DuckDB. But I am not really a data engineer, so perhaps this is naive.