There are 3 Types of Data Engineers.

and you better be the right one.

Then there were three. The final three. Only three.

It’s the truth, there are 3 types of Data Engineers only, and I promise you, you fall into one of the three groups. The truth can be hard, the truth can hurt … but the truth can also heal and lead to growth.

Aren’t you just bubbling over with excitement to know what kind of Data Engineer you are??? Group 1, Group 2, Group 3?

Maybe you’re not happy with where you are at. Maybe you want to move from group 1 to 2, or 2 to three, trust me, you can’t move from 1 to 3 just in a snap … doesn’t work like that.

Group one is always jealous of two, two looks with longing at three. Group three looks down from their high towers onto the peons below them in one and two.

It’s an age-old story.

I encourage you to take your medicine like a champ, no moaning and groaning … no babies here. It’s best to accept who you are, you will be happier in your own skin.

Sure, you can change your group with hard work. I did. I started in group one and went to group two. I thought I had it in me to get to group three, but eventually, I just accepted my fate as group 2 and embraced it.

The 3 Types of Data Engineers.

Remember, I don’t make the rules, don’t be mad at me. It is what it is, and you are what you are.

Without further ado, here are the three types of Data Engineers.

It’s clear to me after working in data for well over a decade that almost all Data Engineers can be put into one of these three categories. Of course, you always have the oddballs that don’t fit the mold, but someone not falling into one of these three slots is rare.

I don’t think any one group of Data Engineer is better than the other, they all have their place, and they are all needed.

• We need people who love SQL, are wizards, and are adept at complex analytics.

• We need people who are good programmers, can build anything, lead projects, and architect Data Platforms.

• We need the Data Engineering savants who are next-level programmers, building the next gen of data tools with Rust etc.

Defining the 3 Types of Data Engineers.

You saw those definitions in the image above, they are what they are. Which one are you? Do you want to be in a different group?

• SQL and Analytics (Group 1)

  • These Data Engineers spend most of their time in SQL, producing analytics and Dashboards. They work extremely closely with the business. They probably don’t write code every day, although some of them are good with Python.

    • The easiest way to transition into Data Engineering.

    • SQL experts

    • Builds data marts, and dashboards, close to the business.

• Senior Level Programmers and Architects (Group 2)

  • This second set of Engineers are the writers of code … complex code. They use a lot of Python, but also Golang, Rust, Scala, etc. They can debug Spark pipelines. They build CI/CD, DevOps, and are comfortable on a Linux server. They build Data Platforms and plan projects.

    • More comfortable on the command line than the UI

    • Broad expert across many tech stacks

    • Understand distributed systems

    • Can design a good Data Platform from the ground up

• Tooling Builder (Group 3)

  • This is the least common of the Data Engineering types, the Yodas of the data space. They build the tooling, private or open source that others use. They are the best of the best, Software Engingeers who specialize in data.

    • Building open-source tooling

    • Expert level programmers

    • Lead/Staff level engineers

I always thought I wanted to be a Group 3, but I had to accept after years of programming that I just wasn’t able to make the switch … I suppose anyone can get there with enough work, and I wasn’t willing to do the work.

I think there be some confusion about Group 2 and Group 3 Data Engineers. One could be a Staff Data Engineer and not really a Group 3.

You might just be really good at building Data Platforms, leading teams, vision, and get to the Staff+ Level. But that doesn’t mean you can build the next Polars. There is a difference.

Inter-group challenges.

Me thinks, and have seen, some sparks flying between these groups of Data Engineers if they get mixed in together … or have to work together.

Engineers and buildings of software (especially in Data) have a high sense of self-worth, they get paid a lot across the board, and generally are proud … which can be a problem.

I don’t think it needs to be this way. There is room for all, and what’s more, there is a need for all. Typically your SQL crunchers using analytics and building Dashboards don’t want to be building the next Pandas tool with Rust.

Many of your Group 2 programmers don’t want to be writing SQL all day to build Dashboards. That’s ok. People are different and want different things.

I think the challenge is when people want to move groups, especially 1 to 2. This can be a slow and tough process with few shortcuts.

My movement between groups.

• I started out many moons ago as a Group 1 Data Engineer. I wrote SQL all day long on SQL Servers and became a next-level expert in the most complex SQL queries possible. After a while, I became tired of all the SQL, Dashboards, and analytics.

• I decided I wanted to be a Group or Type 2 Data Engineer. I wanted to code more. So I simply taught myself.

• I spent a lot of extra time writing code. Learning Spark. Deploying services on Kubernetes. Learning Linux, Bash, CI/CD, DevOps, etc.

• I started to plan and implement bigger projects etc. Eventually, I made the switch to full-time Type 2. It was just what I wanted. Less SQL and more code.

• I’ve been a 2 for years and years. I thought I wanted to be Type 3, but came to realize I just wasn’t a good enough programmer. Sure, I can write Rust, but that’s not the same thing as being able to build Distributed Systems.

I’m super curious my readers … what kind of Data Engineers are you??

I’m also curious about a few questions …

• Are you happy where you are at?

• What group do you want to be in?

• Have you tried to move to a different group?

• Have you had good or bad experiences with other groups?