When I was studying data engineering a trying do use Prefect. I was fascinating because it is simple.... And run in Windows. Airflow takes a long time to config and need to use WSL2 or virtual machine. This is sad. I don't know if Mage can ruin in Windows. I will try it. Thanks for the post. Your article was clear and educational.
Xcom can be a pain in Airflow (to explain to users primarily). Also, the feature is pretty new so buggy but dynamic task mappings leave something to be desired on Airflow as well. Wonder if Mage or Prefect do a better job here.
This is great. I was looking at mage recently as well. One thing one could do in Airflow is to dynamically generate DAGs based on some config. This is commonly used in many home grown config driven pipelines. I do not see how to do that easily in mage.
As always, your content is 🔥 🔥 🔥 . Thanks so much for sharing this 🙏 🙏 🙏
Dagster didn’t get a look in? Any reason why?
Probably because the subtitle says "The Battle for the Orchestration Future." keyword Future
Thank you for steering my attention (curiosity) in the right (maze.ai) direction 👍
Thank you for breaking down your thoughts and experience. Very written as usual!
Thanks!
When I was studying data engineering a trying do use Prefect. I was fascinating because it is simple.... And run in Windows. Airflow takes a long time to config and need to use WSL2 or virtual machine. This is sad. I don't know if Mage can ruin in Windows. I will try it. Thanks for the post. Your article was clear and educational.
Thanks, I think you have missed out on Dagster though, so maybe that would be worth a look as well! I really like it.
Any view on Kestra? I am looking at it vs Mage and Prefect.
Xcom can be a pain in Airflow (to explain to users primarily). Also, the feature is pretty new so buggy but dynamic task mappings leave something to be desired on Airflow as well. Wonder if Mage or Prefect do a better job here.
A lot of people do this easily with: https://docs.mage.ai/design/blocks/dynamic-blocks
This is great. I was looking at mage recently as well. One thing one could do in Airflow is to dynamically generate DAGs based on some config. This is commonly used in many home grown config driven pipelines. I do not see how to do that easily in mage.
I created a framework at Airbnb that does this really easily in Airflow.
I made it even better in Mage. A lot of people do this easily with: https://docs.mage.ai/design/blocks/dynamic-blocks