The Data World Revolves around Partitions.

Times are changing ... are you?

Photo by Juliana Kozoski on Unsplash

Do you ever feel like something just slowly crept up on you, and then it was there, and you have no idea what happened or how it got there? Maybe like getting old. Or last year. Your mom. Whatever. That’s what I’ve felt about partitions.

That topic no one is talking about much, but that is at the core of the new world of Data Engineering we live in. Some SaaS companies talk about partitions, others sorta ignore it and automate it, but in the end, it’s the blood pumping through the veins of many a data platform.

Partitions are found in many tools we use every day, are core to optimal data models, and performant Big Data pipelines. I still don’t know why they are not talked about more, but let’s change that shall we?

Today I want to cover.

• What are data partitions?

• Why we need them.

• What tools use partitions, either forefront or behind the scenes?

• Closing Thoughts.

Thanks to Delta for sponsoring this newsletter! I personally use Delta Lake on a daily basis, and I believe this technology represents the future of Data Engineering. Check out their website below.