So, is data modeling dead?
was it never alive?
I know, probably giving whiplash: one week I'm pontificating on some new tool, the next, like this, writing a thesis on the philosophical question of “Is data modeling dead?”
I've been thinking about this more recently, after talking with Andreas and hearing the strong reactions he has to his belief that data modeling is (dead) overrated. Maybe he's right. But we also have the indelible Joe Reis finishing a book on data modeling, and he's the premier thinker of this data decade.
Truth be told, differing opinions in the data space about pretty much any topic are common and widespread. We all are just the victims of the technologies and jobs/work we’ve been a part of. Data Modeling is as old as the “data profession”
I mean, if the classic “relational model” for data was being written about in 1970, then we are indeed handling a topic that has withstood the sands and flying bytes of time and space. It makes one wonder if in another 50 years, on Mars, someone will be writing another article about the death of data modeling.
Anyways, back to today.
Clearly, there has been an increased conversation around the topic of data modeling over the last handful of years; it seems like COVID got deep into the brain and made us question everything. It seems like the conversation around data modeling has not been how to do it, but whether it is even relevant.
Todays sponsor is Estuary. Without them this content isn’t possible. The best way to support this Substack is to click the links below and give Estuary a try.
Pipelines built by an agent, orchestrated by you
Estuary Agent Skills lets your AI assistant draft production-grade data pipelines from a plain-language description, right inside Claude Code, Cursor, or the tool you already use. The output is config in your repo, not a black box. You review the diff, approve it, version it, and roll it back like any other infrastructure. Same runtime as every other Estuary pipeline. No separate “agent mode.”
Is data modeling dead? Overrated? Will it ever return to its former glory as in the days of yore? Was it ever really alive? These questions have been making the rounds for a few years now; maybe it was COVID that did it.
Even in recent years on Reddit this has come up. Ah … Reddit, methinks it’s the new StackOverflow. A place to get raked over the coals and hangout inside an echo chamber. Yet, I do find myself often scrolling that mucky muck.
Why is the question even being asked?
I don’t know if data modeling is dead or not. Sorry, if that is what you came here trying to find, might as well go back to scrolling Instagram or TikTok, maybe YouTube Shorts if you’re a gray-bearded millennial like me. I will just attempt to present and explore the idea of data modeling, and if it is dead, and let you make up your own mind. Maybe do a survey at the end.
Let’s start by asking the simple question of “Why is this even being asked?”
There has to be some logical explanation, somewhere, of why this has become a topic to even discuss, and seemingly a hot one at that. I believe the answer to this question is fairly obvious. As per this Reddit comment about the reason for data modeling being seen as dead … it’s just become a skill that hasn’t been valued, or at the very least, less valued than it once was.
Why would data modeling have become a less valued skill?
Well, skills rise and fall in time with the popular technologies of choice. Where have we come from the last few decades? SQL Server, MySQL, Oracle, the data warehouse toolkit. Data was SQL. Data was data warehouse. Kimball. OLTP and OLAP. You couldn’t really be a “data professional” in the early 2000’s and before unless you were an expert in data modeling.
The data didn’t live in S3. It wasn’t found in Parquet or JSON files. The data was, by in large, found in relational databases.
Then the data lake happened; the lake house soon followed. Spark arrived on the scene. We wanted better programmers, systems designers, APIs, streaming; the relational databases were, and are, still here, but they moved from the top of the popularity stack to the bottom … and so did those skills … like data modeling. I’m not saying this was right or wrong; I’m just saying this is what most likely happened. People spent time on LeetCode writing Linked Lists and doing palindromes… and less time talking about the finer points of 3rd normal form.
I would suppose this is fairly normal, as technology and abstractions change and morph, the required skills to be successful, or perceived required skills, are going to change as well.
What we are trying to figure out is this.
Is data modeling a required skill to be a data professional today in the average job at the average company?
What does the data actually say?
The only way I know of if we are all just victims of the talking heads convincing us one way or the other, probably because they have a bent one way or the other, if data modeling is dead … would be to look at the average Senior Data Engineering job requirement we can find online, just take a random sample, and found out if compaines are requesting skills like data modeling or data model. I am not saying that all companies are competent and even know what they need, but at the end of the day, we get hired to do a thing. It’s as good a place to start as any other.
So, I went and got myself 30 data job descriptions off of LinkedIn for Senior Data Engineer, or close to it.
I recently did an exercise where I proved to myself I’m not a complete idioit and went back to the old days of writing Rust without an LLM (video and audio processing at that) … since that is how I learned it in the first place. Let’s do the same here to find out out of 30 data jobs, how many require data modeling as a skill.
cargo init datajobs
cd datajobs
cargo add globWe should be able to write a simple Rust program that …
list a directory of files
read a text file
look through text for a word(s)
accumlate a counter for jobs that have that word
Ok, since I’m a Python guy at heart, we will use the glob create.
use glob::glob;
fn main() {
let files = glob("data jobs/*.txt").expect("Failed to get list of files");
for file in files {
println!("{:?}", file);
}
}Yah, Yah, I hear ya, quit messing around and get to the point.
danielbeach@Daniels-MacBook-Pro-2 datajobs % cargo run
Compiling datajobs v0.1.0 (/Users/danielbeach/code/datajobs)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.53s
Running `target/debug/datajobs`
Ok("data jobs/1.txt")
Ok("data jobs/10.txt")
Ok("data jobs/11.txt")
Ok("data jobs/12.txt")Ok, let’s get to it.
use std::i32;
use std::path;
use std::vec;
use std::fs;
use glob::glob;
fn main() {
let files = glob("data jobs/*.txt").expect("Failed to get list of files");
let looking_for = "data modeling";
let mut job_counter: i32 = 0;
for file in files {
let text = read_txt_file(file.expect("bloody bugger"));
let in_there = contain_words(text, &looking_for);
if in_there {
job_counter += 1
}
}
println!("Out of 30 jobs, {} asked for {}", job_counter, looking_for);
}
fn read_txt_file(path: path::PathBuf) -> String {
let contents = fs::read_to_string(path).expect("Dang file, dang it");
return contents
}
fn contain_words(text: String, pat: &&str) -> bool {
let contained = text.to_lowercase().contains(pat);
return contained;
}Nasty little piece of work if I do say so myself. Unlike my friendly local Anonamyous Rust Dev, no on pays me to write Rust. Dang, not what I expected to be honest. All this, including source data is on GitHub if you think I’m lying.
warning: `datajobs` (bin "datajobs") generated 1 warning (run `cargo fix --bin "datajobs" -p datajobs` to apply 1 suggestion)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/datajobs`
Out of 30 jobs, 16 asked for data modelingOver 50% of the jobs asked for data modeling, that is a lot more than I expected, and is a very good sign for all the folk who started out this article by saying in their mind … “You bloody fool, of course data modeling is not dead.” For all you on the other end of the specrtum, it is what it is, not exactly scientific, but hard to argue with. At the very least, one would have to say that if you are in the market for a data job, and you don’t think data modeling is a required skill, than you are severally limiting your pool of jobs. To cut out over %50 of something before you even start is usually not a good sign.
Polling the LinkedIn data professionals.
To make sure I was covering my bases and giving both sides of the argument a fair shake at coming out on top, I decided to post and informal poll, well, not even a poll, but ask the question of my 20k+ LinkedIn followers along with the 37k+ followers on Substack what they think about the topic. I kinda wanted to simply lick my finger, put it in the air and see which way the winds of popular opinion were blowing. I figured it would skew one way or the other, and that is at the very least another good bone to throw on the pile.
Here is the link to the LinkedIn post, and here to the Substack post. I can so that it’s another 80/20 split, most folk affirm that data modeling is an important skill that it is not dead or obsolete.
That’s probably not a suprise to most people, but it was interesting to see the response of folk who leaned maybe more towards what I would call the data modeling is sorta obsolete in the Lake House era. At least from what I could tell reading between the lines. The arguments in this vector seem to stem from a few totally reasonable areas and schools of thought.
New tools are very fast and “cover up” the need for good data modeling.
Small data with no data modeling isn’t that problematic.
Distributed compute with something like Spark and OBT (One Big Table) works well enough.
To be fair, we must give this school of thought some serious attention, because if nothing else, there is a prevading sense in the data community that data modeling is not taken as seriously as it once was. Things like Medallion Architecture are often passed off as “good enough” for a data model in the Lake House, no more work needed.
I have a few thoughts and takes on why we are even having this dicussion, and I think most discussion are valid and worth having. Sure, from what we’ve seen so far, data modeling as a technical skill is far from dead. Over %50 of data job listings ask specifically for that skill, and the powers and thought leaders of our time seem to lean towards the view that data modeling is indeed crucial.
But, to be fair and open, most people would say the same thing about Data Quality and Data Contracts, yet we’ve seen very little to no serious movement in those categories besides white papers and the odd tool here and there. Seen as important but used practically on a daily basis are two different things. So, where and why does this idea that data modeling is dead come from?
The origins of the “Data Modeling is Dead Theory”
I think to explore both sides of this argument fully, we have to park on the data modeling is dead theory for a little bit and try to understand that point of view, and where it comes from. Either way, we can better argue for our viewpoint if we are fully versed on both sides, rather than just propping up our strawman.
I’m going to list some bullet points that I read in online comments to my posts, and others, that are more in favor of the data modeling is dead theory. You can take them as you will.
The last “bible” of data modeling (Kimball’s Tookit) was published in 1996.
15 years ago we lived in a SQL Server, Postgres, Oracle, MySQL dominated relational database world in data land. That is no longer the case.
Many relational database data modeling techniques were designed for performance bottlenecks that no longer exist.
Data technology and compute has changed siginficantly; think DuckDB, Polars, Datafusion, Spark, blah, blah.
The hot choice for data storage is now the Lake House … aka Iceberg and Delta Lake which are file storage systems.
As compared to two decades ago, data has become more complicated in both size, volume, and velocity.
Many people are running Lake Houses with a hodge podge of data modeling, or none at all, and it works*.
Again, you can have a totally different viewpoint on these bullet points, but we must deal with that at some level, argue against them, just try to generally understand them and how to fit into our metal model that data modeling does indeed hold an important part of the data platform. Back in the day, our Data Warehouse exsisted on SQL Server and if our data model sucked, our queries didn’t run, the reports were slow, and you would get fired by the CTO.
Today you can have One Big Table sitting in parquet files on GCS, point Big Query at them, and get the answers you need well enough. It might not be cost effective, but it’s possible the CTO as such an organziation could care less, as long as that line item stays in the same spot of the budget and doesn’t get out of hand.
Does the data model need to change with tecnology?
This is a valid question that comes up often as well. Does the data model; Kimball, Imon, dreamed up decades ago need to be adjusted for the new world we live in? I’m not saying yes or no, I’m just asking the question. I’ve lived through these data changes, I cut my teeth on data teams running the Data Warehouse on SQL Server clusters, where we tuned every index and query, and were manical about the data models at every layer.
We didn’t have a choice, the whole ship would slow to a halt if we screwed up the data model.
No one has become the data modeling messiah of the 2000’s. It’s been an empty void of talking heads regurgitating the OLTP and OLAP bullet points we learned from our forefathers. When is the last time I read an article with some data influncer going over the finer points of Third Normal Form? Yeah … doesn’t happen. So, if we want data modeling to be taken seriously, then the data community needs to teach it and apply it to the Lake House archteciture so the data engineers are not left in the dark to feel their way about the bits and bytes finding their own way.
What if data modeling is more about the business (than the technology)?
For someone like me who grew up with the Data Warehouse Toolkit next to my literal Bible on my nightstand, I feel like it’s possible the entire data modeling conversation, in all its technical glory around DuckDB, Spark, Databricks, Snowflake … or whatever, is missing a core componet.
This componet rarely gets brought up in the argument from the “Data Modeling is Dead” crowd. That part is … The Business. I would like to quote Joe Reis who’s written extensively on this topic and is the defacto expert of our age.
“We Model the Business, Not Just Data Systems”
- Joe Reis
This has always been the problem of engineers who focus to much on the technical aspect of any topic, not that any of those technical things are not important. It’s just that to create good Data Platforms and code, we must remember we are here to serve “the business.” Everything we do happens within a context. All those annoying data models are only half for tools, the other half are for the … PEOPLE. Us. Me. You. The collective WE.
We model our data after business logic to reduce errors, help humans and AI trained on humans, better understand context, answer questions, and make sense of the unknown using good data models.
Good data models increase accuracy, reduce errors, and generally reduce the burden on humans and agentic solutions tasked with bringing answers out of the chaos that has always been data.
Is data modeling overrated?
Beauty is in the eye of the beholder, it depends on the context of you, the company you work at, the tools you find yourself using every day. I think it’s a sliding scale. Maybe you find yourself in Postgres or SQL Server, than of course you’re going to have a high view of data modeling. Maybe you’re working at some marketing company who dumps stuff into JSON files on GCS and just throws BigQuery on top and is off to the races. Maybe they don’t want you to spend time data modeling.
I guess you have your answer then.
But, maybe you’re working on a Databricks or Snowflake platform, your data is large, there is a new wave of agentic solutions that need to be given access to a complex and vast array of data. Than the data modeling is probably going to make or break your data platform success.
Can you get away with not data modeling “very much.” Yeah, if you are using Spark ontop of Iceberg with just a handful of tables, yeah … you will probably survive … but not thrive. Maybe you have 100GB of data and you are using DuckDB and Polars ontop of Parquet files in cloud storage, and it’s just easier to and faster to get the answers the way the data landed.
Sure, it probably works, as long as you have 5 reports that no one looks at.
Again, you get out of your data what you put into it.
If you spend time to curate your data, understand it, massage it into meaningful formats and models, yeah, you’re probably in the top 10% and you get returns on your data modeling investment. I myself eat my own words, today, I work in a place were we made the concious decision to run a Kimball style data modeled Lake House with Facts and Dimensions as if it was SQL Server form 30 years ago. That was probably just my pre-concieved notions and background playing into it.
People who call data modeling dead, overrated, etc - FAFO
Whenever someone states that some technology or practice is dead, I'm reminded of a G. K. Chesterton quote that no one should be allowed to remove a fence unless they know why it was built to start with.