Behold. ShortCircuitOperator. Apache Airflow.

I was today years old when ...

It’s challenging to keep up with the rapidly changing landscape of tooling, let alone the evolving features and updates across the broad spectrum of Data Engineering tooling.

Apache Airflow is no different.

We use it. We love it. Once every year or two, we might upgrade a version if we are feeling spicy.

Truth is, just like SQL, Spark, or whatever … we fall into, or learn to write DAGs in a certain way and simply move on with life and Data Engineering, we work on more interesting things.

This sums up my relationship with Apache Airflow. Use it, abuse it, and mostly ignore it, writing my DAGs in the same way I learned to do so many years ago.

This is good and bad. Good in the sense that a tool like Apache Airflow is so mature and well-oiled that you don’t have to worry much about breaking changes; you can learn the fundamentals and move on.