Comment by dogman123
3 days ago
i'm working on a project to do this with iceberg and sqlmesh executed via airflow at my job. sqlmesh seems really promising. i investigated multi-engine executions in dbt and it seems like you need to pay a lot of $$$ for it (multi-engine execution requires multiple dbt projects) and is not included in dbt core.
Toby and the team at Tobiko really are a pleasure to work with. They have strong opinions but have shown a good amount of willingness to implement features as long as there's a strong general use case. We've been working with them for almost a year now and it's really interesting seeing a early-ish open source library being developed by a start-up develops (and how much influence you can have over the direction if you work closely with the dev team).
that's great to hear. it mirrors my observational experience from being in their slack channel. i'm aware of the technical risks of being an early adopter of a product like this, but i must say part of me is excited to be on board early to help to shape it from a user perspective. i'm still not totally bought in yet (still in mvp phase) but our use case as we scale almost requires multi-engine execution (athena, spark on EMR, duckdb) and it doesn't seem like anyone is doing it better.
I'm one of the cofounders at Tower, posted this because I thought some people would be interested in the topic. Would be interested to know what Airflow is really...doing...for you here? Is it just an execution engine for your sqlmesh? Anyway, as we're trying to build out Tower would love to know more.
sqlmesh execution engine + cloud resource provisioning
provision spark on emr or duckdb on beefy ec2 -> run sqlmesh -> wipe resources.
i'm still in MVP phase of revamping my company's current data platform, so maybe there are better alternatives -- which i'd love to hear about.
Cool, would be happy to talk a bit more about it. Want to shoot me an email (in bio) or if you sign up for our waitlist on https://tower.dev I can reach out in the AM (we’re in Germany)!