Data Mesh Theology. Dead or Alive?
dying on the vine?
Meh. That's how I feel about the languishing and short-lived ideology called “Data Mesh.” The Modern Data Stack has seen its fair share of prophets seeking the loving embrace of the community, a fickle thing, only to fall out of favor all too soon.
Data Mesh appears to be one of those ideas.
The thing is, I keep my trained ear close to the ground, listening to the comings and goings of various and sundry ideas. Watching what becomes popular, what dies on the vine.
But I won’t be the judge of that in the end. Never say never. Let’s just lay it all out.
Try to understand what a Data Mesh architecture is
How long has it been around
Pros and cons
Is there any hope left for mainstream adoption?
This is just as much for me as for you. I have my preconceived notions, of course, but we will try to give Data Mesh a fair shake and see where it's come from and where it might be going.
You know one thing is true … those cloud compute bills are getting out of hand. Check out https://carolinacloud.io/ for High-Performance CPUs, GPUs, Notebooks, and Bare-Metal Servers.
If you want to save serious money on compute, head over to Carolina Cloud.
What is Data Mesh? (In Plain English)
Data Mesh is an architectural approach that aims to address the problems of large, centralized data teams by decentralizing data ownership. Instead of one monolithic team owning all datasets, each domain (fraud, marketing, product, etc.) owns its data as a product.
This can best be explained visually, contrasted with the most common approach of gathering all data into a single spot to be shared from there.
Let me give you a few bullet points to further drive home what a Data Mesh Architecture is.
decentralized data architecture,
domain-oriented
self-serve design
“Responsibility for analytical data is shifted from the central data team to the domain teams.”
This is in direct opposition to the classic approach of a single Data Platform team handling ALL data for an org, ingesting all data, doing their work, and becoming the central point of contact and distribution.Data Mesh is the reverse of that approach. It treats each dataset as a product that is handled, produced, and served … by the data producer.
Where did Data Mesh come from?
I think that it can be helpful to understand a concept by examining where it came from, how it’s morphed over time, and so on. Seeing what the people who made a thing are up to can give us insight into where something might be going.
So, let’s look at where Data Mesh came from.
“The term data mesh was first defined by Zhamak Dehghani in 2019[7] while she was working as a principal consultant at the technology company Thoughtworks.[8][9] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[10][11] Data meshes have been implemented by companies such as Zalando,[12] Netflix,[13] Intuit,[14] VistaPrint, PayPal[15] and others.
In 2022, Dehghani left Thoughtworks to found Nextdata Technologies to focus on decentralized data”
You can find Zhamak today working on NextData.
NextData OS is apparently the final regurgitation of the Data Mesh theology.
“Nextdata OS - Autonomous data product platform for agents, machine learning and analytics. Turn any source into data products that run and govern themselves autonomously: composable, AI-ready, can ingest any format, run on any stack and serve any application.” - NextData
“… make an observable mesh of data products across your enterprise.”
Reading the NextData website, you will find a few mentions of Data Mesh, but it definitely is not the main thrust per se. I’m not sure what to think about that … well, I do know what to think, I will leave the rest to your imagination.
Pros and Cons of a Data Mesh architecture.
I think we can all agree, at a high level, that there are many benefits to the Data Mesh philosophy. On the surface, it’s not that hard to see the good in it.
Pros
- Very large organzations could benefit from forcing data ownership
- The bottle neck of a single Lake House for ALL data could be removed
- Data sits and resides in the domain that knows it best
- Forces self serve Data Platform/Analyticsvs
Cons
- Cannot be resonable adopted at even medium sized companies
- Domains don’t want to own their data
- Domains don’t have know-how to manage their data
- Governance becomes harder, not easier when everything is decentralized
- Tooling never matched visionI mean, back in 2023 places like PayPal mentioned Data Mesh as their architecture of choice, so you know it has real value as an idea within a large and complex organization.
I polled my LinkedIn and X folk about Data Mesh to see if it was still alive and well in the wild, and got a mixed bag of responses. A whole lotta nothing on LinkedIn, but some positive feedback on X that it is indeed a thing in some circles yet.
Yet others are clearly dissatisfied with Data Mesh and migrating away due to “high costs” and “data silos.”
Heck, even Gartner created some buzz by saying that Data Mesh was going to die out. Depends on how seriously you take them.
Clearly, there are some strong headwinds against Data Mesh, which is struggling, real or not. The general perception, aside from a few oddities, is that Data Mesh was good in theory but lacked seriousness in practice.
So, if big tech has always been a North Star for many techies, why has Data Mesh been so silent lately? Why has it never been talked about in r/dataengineering except for people asking if it's still alive?
The truth is that only seriously technical teams and companies can implement a Data Mesh in a way that overcomes the downsides of that architecture. Clearly, orgs like Netflix are capable and willing to do it, and do it well.
But, at the end of the day, if only a few select orgs have done it, and no one in the community at large even mentions the words Data Mesh for a few years straight, it’s going to be hard to make a real argument that Data Mesh is anything but a fringe architecture that seems to be dying on the vine.
Clearly, this could just be one person’s take, but I think it’s more than that. If we put our fingers and noses to the wind, it’s fairly easy to make reasonable judgement calls using the obvious writing on the wall in front of us.
The Real Take
Data Mesh wasn’t wrong—it was just misaligned with how organizations actually operate. I also want to be clear, we are talking about the 90% of organizations that simply don’t have the size to pull off a Data Mesh architecture.
Most companies are not of size.
Most companies need:
strong centralized platforms
collaborative data modeling
clear ownership
and consistency
… not each domain reinventing its own data stack. It takes serious chops to have experts in data scattered across teams, doing whatever they want with technology and approach, and defining business terms. It’s hard enough on a single Data Platform.
Do you force everyone to use the same technology? How are they trained? Do you have a centralized Data Executive?
But, on the flip side you can find a few examples of sucessful Data Mesh implementations at large compaines, that seems to have a good experience.
I mean, you can find some vendor blogs, like Snowflake, talking about how to do a Data Mesh with “their tool,” but as far as general industry knowledge, writing, talks, and tools specific to Data Mesh, there just isn’t a lot going on.
It’s a problem of semantics and tools
With all the time I’ve spent researching Data Mesh and its shaky history of starts and stops, I just can’t help thinking that Data Mesh was never actually alive, in any real sense.
Could the future prove me wrong? Maybe, it’s hard to know the fickle flow of the data commuity at large, and where the tides will go in the future.
At the end of the data, a few things are clear to me.
Data Mesh is an idea, not a technology
Data Mesh is for “large” companies
It was only ever sparsely adopted
It lacks any consistent serious development (unlike Lake House architecture)
It works great for some people
Others only have horror stories
Based on comments from all around the interent, most of the experiences are not good, but it appears to be highly dependant on the technical chops and the commitment of said organzation. That has more to do with the business, and less of a judgement on Data Mesh as an architectual design.
Let’s be truthful, in the world of cost-cutting, layoffs, and the like, if a Data Executive looks at both the traditional Data Platform approach and the Data Mesh approach, what are they going to pick??
You need at least a Data Engineer and a Data Analyst for each “Data Product,” and most likely a Data Scientist to boot. Where do the middle management and the Data Leaders sit? At each product, nowhere, everywhere??
If you argue that you can have a single Data Platform, aka say Databricks, yet apply Data Mesh architecture and give the power to each Data Product/Business Unit line … then …
Who runs maintenance and oversees the Data Platform as a whole?
Who makes the executive-level decisions on data strategy?
How does alignment happen?
Who’s in charge of bringing all together for Analytics and DS use cases?
If your answer to any of those questions is some set of person(s) or department that makes over-arching decisions … then that flies in the face of one of the MAIN arguments for Data Mesh architecture. To get away from those single bottlenecks.
I haven’t been unclear, have I? I hope not. Personally, I think Data Mesh is a bad idea for most orgs, though I have to tip my hat to the few large and brave who have pulled it off and like it.
You can’t argue with results.
If someone says they love Data Mesh, it solved many of their problems, what’s a brother to say? Nothing. Just tip of the hat to ya.
We live in a large world with lots of people who don’t do all sorts of things. Heck, we got people running Oracle Data Warehouses to their heart’s content, ignoring the words “Lake House.”
Anything and everything is possible. But not everything is advisable.
The glass-half-empty dude in my says that after decades of watching single Data Teams fight and claw to provide valuable and correct insights, while not killing each other, it is hard enough. Let alone 10 different teams trying to align on business values, concepts, technology, and definitions.
Seems like a Pipe Dream, but apparently even Pipe Dreams can come true every once in a while.
You know one thing is true … those cloud compute bills are getting out of hand. Check out https://carolinacloud.io/ for High-Performance CPUs, GPUs, Notebooks, and Bare-Metal Servers.
If you want to save serious money on compute, head over to Carolina Cloud.
This is a very good read. I would say that for Data mesh kinda architecture to work properly, there needs to be a central data platform team who takes care of key pieces like tools for teams to do ingestion, domain segregated infra modules, enables org wide central data catalog, provides tools for federated data governance. For domain teams to function in an aligned manner, platform team should be the one to enforce what, where and how pieces of the platform. At the end, if domains don’t want to own this way of functioning, then this can’t survive.