Comment by geysersam
3 days ago
I've been looking at dbt for exactly this reason but don't quite get the advantages if you're not interacting with a data warehouse of some sort.
3 days ago
I've been looking at dbt for exactly this reason but don't quite get the advantages if you're not interacting with a data warehouse of some sort.
Spark/PySpark/polars/dbt/sqlmesh are all data engineering frameworks for data transformation (and some data scientists).
If you don't have a data warehouse / OLAP system you are generally not in the niche for those tools.
Embedded databases like DuckDB are just as suited for dbt work.