After using Argo Workflows, I don't think I will ever return to Airflow. Kubernetes is not an easy system to manage, but managing an Airflow setup is somehow worse. The story around disaster recovery and scheduler redundancy was an absolute nightmare for me.
Or worse, airflow!
Airflow can be frustrating but when it works it is so satisfying.
After using Argo Workflows, I don't think I will ever return to Airflow. Kubernetes is not an easy system to manage, but managing an Airflow setup is somehow worse. The story around disaster recovery and scheduler redundancy was an absolute nightmare for me.
2 replies →
I think mistaking Airflow for a mere "task scheduler" is part of that frustration.