← Back to context

Comment by otter-in-a-suit

1 day ago

Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a giant several hundred step dbt pipeline" to "query the output of said pipeline" to "read/query a csv on disk" with the exact same tool is just so nice. If I could centrally manage said asset more akin to a traditional database, I'd be very happy.

I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I really like way too many things about duckdb to replace it.

[1]: If you care: https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spati...

18 comments

otter-in-a-suit

Reply
mediaman  21 hours ago

I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop), but there's so much I like about it, and it's really nice that each tenant can have total separation with custom schemas and that it's straightforward to pile data into object storage and form your own lake.

And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.

  • ljosifov  3 hours ago

    Similar. Noticed DuckDB ever since an old article 'what db should I use' for local small data warehousing. The author was blown away that DuckDB seemed super naturally quick. It was I think columnar store + compression facilitated that? It made duckdb load compressed + on the fly decompress = faster than even reading the uncompressed data. Had forgotten most of it. But was used to mmap-ed files + columnar storing of Kdb. Was pleased that the author was clued to notice the power of that.

    Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)

    Latter I read this https://openai.com/index/inside-our-in-house-data-agent/ and thought "but of course".

  • allan_s  19 hours ago

    As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ?

    The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?

    • mediaman  17 hours ago

      Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine).

      But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).

      With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".

      3 replies →

    • otter-in-a-suit  13 hours ago

      There's a few options.

      Clickhouse, as I mentioned, can be a good final layer, as can postgres.

      You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.

      duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).

      All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.

      For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.

    • 8note  14 hours ago

      another variant:

      i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it

      is your setup running into problems that makes you need something more?

      1 reply →

  • tccole  21 hours ago

    I feel like familiarity and ease of use and “good enough” beat out the perfect db for the job in many occasions.

dmix  20 hours ago

> to "read/query a csv on disk"

I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.