Comment by Kydlaw
3 days ago
Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
I've been using clickhouse-local for quite some time, instead of DuckDB. There is also chDB.
After using pandas for 10 years, I favor SQL now, for some reason.
I haven't tried chDB yet, but I heard about it. Thank you for reminding me of that option.
I use SQL in data pipelines and processing that is going to require interoperability.
But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
Maybe because it's like a swiss army knife for data work, regardless of whether you need it for OLTP or OLAP workloads. Having different SQL dialects is a bit annoying, but the base is the same more or less, so switching doesn't come at too big of a cost.
As a huge duckdb fan, I'd love to see chDB to get proper windows support - that would make it real competition (having WASM coverage is already a big step) which would be good for the space as a whole.
Interesting, why do you typically prefer clickhouse local to duckdb?
Because we use regular CH in production. I'm pretty familiar with it and I can always run large queries on a production server with 1TB of RAM.
Regular CH also support external data sources, so I can read 500GB of JSON from S3 and group by it on production server very fast and in memory.