Comment by Fervicus
8 hours ago
Thanks, I gathered that much. I am curious what kind of stuff people are using it for, especially in their personal projects.
8 hours ago
Thanks, I gathered that much. I am curious what kind of stuff people are using it for, especially in their personal projects.
One great use case I've found:
You can find monthly archives of every reddit comment on Academic torrents. These are huge NDJSON files compressed to .zst, named like 'RC_2026-01.zst'. The size is ~60GB compressed, 350GB+ uncompressed, per month.
Most of the size is taken by the actual comment text. But I was only interested in calculating how many unique commenters subreddits have in a month so I only wanted to extract a few fields from it and discard the rest.
If you use traditional tools like pandas or load the data to a database and then query it, you would quickly run out of RAM or storage space, especially when doing it on a basic laptop like I was. But with DuckDB it's just this:
``` SELECT lower(subreddit) AS subreddit, author FROM read_json(['RC_2026-01.zst', 'RC_2026-02.zst']) ```
DuckDB automatically handles decompression on the fly, figures out the schema, manages RAM so you won't OOM and so on. And even on my laptop that query takes like, 2 minutes? Which is super impressive to me.