Debugging Data Pipelines.
Is it a pipeline issue ... or a data issue? Or both? Argg.
Some things never change, and you would think as the years pass by, debugging software would become easier. I have found that data pipelines seem to have a “special” kind of hard, insidious bugs that reap a harvest of sorrow upon Data Engineers.
Debugging data pipelines is probably one of Dante's seven rings of hell. Sure, you can do it over and over again, but complex data pipelines have one big problem. Data. Data is messy, data isn’t what you think, data changes, and data adds an extra layer of complexity. Data is hard.
The lethal combination of code and data will bring the Senior Engineer with decades of experience to their knees. But, there is light at the end of the tunnel. Fortunately, there are things you can do to lessen the pain. That’s what we are going to cover today.
What makes debugging data pipelines hard?
How to prevent data pipeline bugs.