Comment by lysecret
6 hours ago
Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing.
6 hours ago
Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing.
What I don't understand is why they didn't build this project on top of DuckLake instead - from my shallow analysis they vibe coded up a fake Databricks in Python.
Not that I'd use it, I want to avoid Databricks - the Duck ecosystem is far more compelling for my modest needs, and handles local develop a lot better (IMO).
Ducklake works well. So far it is still under active development (post AWS acquisition). There is support for local disk-based and remote S3 based datalakes, so integration tests become really easy. Seems stable and all the bugs I've found have already been patched.
Any risks in terms of using it on gcp/azure?
Praying for the poor Ducklake employees at AWS to keep whatever demon works there at bay who insists that every single data product has to be bolted onto AWS Glue (and its various Data Catalog, Data Lake, etc. partners in crime) in the most clumsy way possible. And please keep it away from Athena, possibly the worst major query engine in terms of support for query features and file formats. I don't want to see AWS-era Ducklake become another failure like S3 Table Buckets are (imo at least).
I'd personally love to try this in earnest as well