Atomic Transactions in Databricks Spark SQL
for Delta Lake and Apache Iceberg
Sometimes I’m not sure whether I should just sell all my belongings, move to a cabin in the woods, and raise goats, or keep this Data Engineering thing going a while longer. The problem is, if you stay too long (like me), you're liable to start seeing history repeat itself a little bit.
Methinks we should basically put all data folk into one of two baskets, for easier counting and general relatabiliity. Sorta like a nursing home of sorts for the data community.
I should have seen this one coming, I didn’t, but I should have. With announcements and features like Stored Procedures on Spark/Databricks hitting the block, only an idiot like me would not have seen “transactions” coming down the pike.
A sure sign of my age and senile ways.
Well, all you youngsters sucking on the teat of Apache Iceberg and Delta Lake might say, “But we already have transactions in our Lake House architecture. We do INSERT, UPDATE, and DELETE while you’re still stretching your old bones in the morning.” Indeed, my friend, you do have something like a transaction.
But all those decrepit DBAs turned Data Engineers know better. I see you.
