It's never quite been clear to me what the advantage of dbt over a python program using sqlalchemy / duckdb / polars to transform data is. Can you enlighten me?
At the minimum, it's just Jinja2 templates in your SQL queries - meaning, you can do pure SQL transformations with conditional logic in your templates. In addition to being able to run tests, specify custom macros, having version control and having some constrained way to organize your tables, you're turning SQL into a proper programming language with just one library.
My issue with DBT is it is a mix of SQL, yaml, jinja2 flow controls (and metrics is whole another thing). SQL with jinja2 if/else can get really unmaintainable quickly. It's perhaps better than homegrown sql based transformers.
polars is code and can be version controlled too. Dataframes in my opinion are more elegant, and with the right backends and some lineage enhancements, could serve a much wider set of use cases than what DBT does
It's never quite been clear to me what the advantage of dbt over a python program using sqlalchemy / duckdb / polars to transform data is. Can you enlighten me?
At the minimum, it's just Jinja2 templates in your SQL queries - meaning, you can do pure SQL transformations with conditional logic in your templates. In addition to being able to run tests, specify custom macros, having version control and having some constrained way to organize your tables, you're turning SQL into a proper programming language with just one library.
My issue with DBT is it is a mix of SQL, yaml, jinja2 flow controls (and metrics is whole another thing). SQL with jinja2 if/else can get really unmaintainable quickly. It's perhaps better than homegrown sql based transformers.
polars is code and can be version controlled too. Dataframes in my opinion are more elegant, and with the right backends and some lineage enhancements, could serve a much wider set of use cases than what DBT does
1 reply →
But is that ... good?
3 replies →
I've been looking at dbt for exactly this reason but don't quite get the advantages if you're not interacting with a data warehouse of some sort.
Spark/PySpark/polars/dbt/sqlmesh are all data engineering frameworks for data transformation (and some data scientists).
If you don't have a data warehouse / OLAP system you are generally not in the niche for those tools.
Embedded databases like DuckDB are just as suited for dbt work.