A story as old as time, you and I’ve been around the block plenty before eh? Yet, after more than a decade of doing Data Engineering work … I still forget sometimes. Old habits die hard.
If I think back even further, to my old SQL Server Data Warehouse days, the same thing applied. I spent years tuning queries, and even today, piddling around Spark pipelines looking for a little efficiency here and there.
It comes down to one thing … filter early, filter often.
Keep reading with a 7-day free trial
Subscribe to Data Engineering Central to keep reading this post and get 7 days of free access to the full post archives.