13 Comments

As always, your content is 🔥 🔥 🔥 . Thanks so much for sharing this 🙏 🙏 🙏

Expand full comment

Dagster didn’t get a look in? Any reason why?

Expand full comment

Probably because the subtitle says "The Battle for the Orchestration Future." keyword Future

Expand full comment

Thank you for steering my attention (curiosity) in the right (maze.ai) direction 👍

Expand full comment

Thank you for breaking down your thoughts and experience. Very written as usual!

Expand full comment

Thanks!

Expand full comment

When I was studying data engineering a trying do use Prefect. I was fascinating because it is simple.... And run in Windows. Airflow takes a long time to config and need to use WSL2 or virtual machine. This is sad. I don't know if Mage can ruin in Windows. I will try it. Thanks for the post. Your article was clear and educational.

Expand full comment

Thanks, I think you have missed out on Dagster though, so maybe that would be worth a look as well! I really like it.

Expand full comment

Any view on Kestra? I am looking at it vs Mage and Prefect.

Expand full comment

Xcom can be a pain in Airflow (to explain to users primarily). Also, the feature is pretty new so buggy but dynamic task mappings leave something to be desired on Airflow as well. Wonder if Mage or Prefect do a better job here.

Expand full comment

A lot of people do this easily with: https://docs.mage.ai/design/blocks/dynamic-blocks

Expand full comment

This is great. I was looking at mage recently as well. One thing one could do in Airflow is to dynamically generate DAGs based on some config. This is commonly used in many home grown config driven pipelines. I do not see how to do that easily in mage.

Expand full comment

I created a framework at Airbnb that does this really easily in Airflow.

I made it even better in Mage. A lot of people do this easily with: https://docs.mage.ai/design/blocks/dynamic-blocks

Expand full comment