Comment by otter-in-a-suit

There's a few options.

Clickhouse, as I mentioned, can be a good final layer, as can postgres.

You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.

duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).

All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.

For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.