Postgres to Delta Lake ... and back again.

back and forth

Strangely enough, or maybe not so strange, when you find yourself working in a Lake House environment, it’s not uncommon to find the odd Postgres database hanging around the edges. You know, like the third wheel at the party in the corner looking awkward.

I think it’s one of those things; the whole RDMBS ← → Lake House is something everyone does, and no one talks about for some reason. Too dull, too many ways to do it? Who knows.

On a semi-regular basis I find myself doing one of two things.

• pushing data from Delta Lake into Postgres

• pull data from Postgres into Delta Lake

I mean, it’s the classic Data Engineering problem, right? Running old and new technology beside each other makes them play nice together!

Very often, our data systems as a whole are made up of various pieces and parts, and the Lake House + RDMBS is one that often crosses paths.

So today, we will examine this problem and try to solve it using several different technologies. We will see which one we like best.