Comment by phainopepla2
12 hours ago
DuckDB is for analytics, so OLAP instead of OLTP. If you're using the DB for normal transactions, stick with sqlite
12 hours ago
DuckDB is for analytics, so OLAP instead of OLTP. If you're using the DB for normal transactions, stick with sqlite
Thanks, I gathered that much. I am curious what kind of stuff people are using it for, especially in their personal projects.
One great use case I've found:
You can find monthly archives of every reddit comment on Academic torrents. These are huge NDJSON files compressed to .zst, named like 'RC_2026-01.zst'. The size is ~60GB compressed, 350GB+ uncompressed, per month.
Most of the size is taken by the actual comment text. But I was only interested in calculating how many unique commenters subreddits have in a month so I only wanted to extract a few fields from it and discard the rest.
If you use traditional tools like pandas or load the data to a database and then query it, you would quickly run out of RAM or storage space, especially when doing it on a basic laptop like I was. But with DuckDB it's just this:
``` SELECT lower(subreddit) AS subreddit, author FROM read_json(['RC_2026-01.zst', 'RC_2026-02.zst']) ```
DuckDB automatically handles decompression on the fly, figures out the schema, manages RAM so you won't OOM and so on. And even on my laptop that query takes like, 2 minutes? Which is super impressive to me.