Batch vs Near-Realtime vs Streaming

Nothing New Under the Sun

I feel like if we all listened to the blithering pundits on social media for the last few years we would have gathered all batch ETL pipelines in a pile and burned them all till the flames reached the skies and we all danced around in the moonlight till we frothed at the mouth and collapsed into a pile.

I’ve been hearing about the death of Batch Data Pipelines for years now … usually, it comes in conjunction with the release of some new streaming tool.

You would think people would learn their lesson by now. 90% of Data Teams will never need to use Streaming Pipelines, no matter how much you try to make them cool. They are simply unneeded… for many reasons.