Comment by sureglymop
6 hours ago
I've used Polars before and can only recommend it.
It's like you get a really good 'query planner' like a DB would give you, but for your notebooks/scripts/etc. Much better than pandas imo.
6 hours ago
I've used Polars before and can only recommend it.
It's like you get a really good 'query planner' like a DB would give you, but for your notebooks/scripts/etc. Much better than pandas imo.
Why use Polars over just postgresql or sqlite?
It is optimized for analytic workloads (default in-mem columnar layout), whereas PG and sqlite are OLTP DBs. It's going to be insanely faster for those use cases.
And with DuckDB?
DuckDB is an in process OLAP, I’ve been using it a lot, and I am keen to use Polars but DuckDB seemings to be flying for me.
Actual databases pay up to two orders of magnitude of speed for durability. If you can regenerate your dataset in case the DB drops it (or there is a power outage, or whatever), being able to complete a complex, write-heavy query in 10s instead of 15 minutes is actually very useful when doing analysis.
Because those are databases? Polars is a data processing engine, not a database. They have different uses.
If you want to use a database instead of Polars, for similar use cases, duckdb is a much better bet than postgressql or sqlite.