Designing BETTER Data Pipelines.
Become the master of Data Pipelines with a few easy tricks.
“It's funny after all these years, millions of data pipelines plugging along in quiet benevolence for decades. Grab some data, load some data, transform some data, rinse and repeat.”
If it's so mundane then do we need Data Engineering at all, why are Data Engineers in such high demand, even in these uncertain economic times? Why hasn't true no code come to fruition yet? Why are so many data pipelines at this very minute, breaking and puking errors and alerts, waking up slumbering and sleepy eyes Data Engineers to look gloomily at glowing computer screens while punching the keyboard with a frenzied frustration that has built up over the years?
“I mean haven't we done this enough by now to build every data pipeline to be invincible? Nope. Death, taxes, and data pipelines.”
Today we will end the despair and recover as if from some ancient scroll, the fundamental truths to designing BETTER data pipelines. Wizened and dusty, we are going to breathe life back into these old ideas, and see if they contain any magic and stardust that can save us from ourselves.
