Data Engineer vs Senior Data Engineer

What is the difference between a Senior Data Engineering and a Data Engineer?

Engineers at every level and in any context are always asking some of the same questions, like what it takes to move to the next level. As humans alive on this planet, we are all cursed with a sort of self-deception, our own self-perception as it relates to our skillsets and skill level.

I have struggled with this often in my career. As a self-taught developer, it took me a long time to overcome imposter syndrome, fighting and striving to overcome that issue, and easily falling too far on the other side of the fence. Overestimating and seeing myself as superior in areas that I still need work in.

It’s like grading your own papers.

What do you see when you look in the mirror? How do you know when you are ready for the next level? What is the difference between a Senior Data Engineer and Data Engineer?

The difference between Senior DE and DE isn’t what you think.

I can cleary remember a time in my life when I wanted the Senior Data Engineer title very badly. But that didn’t mean I was ready for it. Of course when you want something to happen, you will pretty much tell yourself and convince yourself of anything.

There are a few common misconceptions that more junior engineers have about becoming a Senior Data Engineer.

• It’s purely technical skills that separate Senior from not-Senior level.

• You have to be an expert on every new or popular tool released.

• You have to be a star programmer, agile, TDD, OPP, functional genius.

• You need to be able to code in Java, Scala, and Python.

• Doing good work on time is enough to qualify you.

In my experience, although there is nothing particularly wrong with anything in that list, and they are all worthy goals to be pursued, in most cases it is not these things are are the separation between a Senior Data Engineer and a Data Engineer.

In fact, quite the opposite is true, and the skills and value add is at the other end of the spectrum.

What Senior Data Engineers bring to the table.

So if it isn’t a never ending list of amazing technical skills that make a Senior Data Engineer bring enough value to be given that title, what is it?

Here are a few of the skills and characteristics that I have seen Senior folks display consistently over time, and what makes them great team members and core drivers at the heart of Data Teams.

• Able to drive a project autonomously without much need for guidance.

• Able to upskill other team members.

• Able to delegate tasks, not micro-manage and let others learn through trial and error.

• Able to learn new technical skills and tools quickly when needed (adaptability).

• Easy to work with, kind, understanding, and empathic.

• Can make hard decisions and decide on a direction.

• Ability to admit mistakes and course correct.

• Breadth of knowledge on many topics (architecture, devops, ML, pipelines, SQL, etc.) ( I didn’t say absolute knowledge.)

• Know what corners to cut, and which one’s should not be cut.

• Aligned with the business and product, understanding what’s important for them.

• Works hard to write good, clean code.

“Senior Data Engineers excel in many non-technical areas that help them provide value to the business and fellow team-members. Their technical skills in themselves are just icing on the cake.”

Let’s quickly dive into each one of these areas and see why they are such important skills that make the difference between a Senior Data Engineer and someone who is not.

First, I want to mention something. It’s not that technical skills are unimportant, it’s just that they are a given if you’re at a Senior level. You should be writing unit tests, you should be writing concise clear code. Just being good at those things makes you a good individual contributor, if you want to multiple your impact, it takes something more.

Able to drive a project autonomously without much need for guidance.

This is probably one of the top non-technical skills any Data Engineer can gain, the ability to be given a task, project, epic, whatever, and manage that project well, defining units of work, understanding pain points, and drive that project through to the end without a lot of oversight.

So many skills are built into running a project, communication and understanding with business priorities, what’s important, what’s not, delegation, planning, and the list goes on.

Guidance takes time and energy, it’s great to give this type of direction to Engineers, but if you’re a Senior Data Engineer, you should need less of this. You should be able to run projects and make decisions yourself.

Able to upskill other team members.

Closely related to the last topic, the ability of a Senior Data Engineer to upskill other members of a Data Team is critical to the smooth function and efficient processing of projects and tasks.

• Upskilled team-members stay around longer.

• Upskilling engineers improves relationships and communication.

• Upskilling engineers helps Seniors dig into what they are teaching and why.

As well, I’ve found that upskilling other team members and lead to a lot of learning myself. Being challenged on some implementation, and looking at problems from a different perspective is the best way to learn and grow as a Senior as well.

Able to delegate tasks.

Nothing will slow down a data team to a painful crawl like someone who thinks they must work on everything themselves, or “saves” the most important and difficult tasks for themselves.

The belief that you are the only one smart enough to complete a task correctly is very harmful to others and yourself. It’s impossible to grow as a Data Engineer if you aren’t challenged with new problems to solve.

“Every Senior Data Engineer should actively seek to delgate difficult and important tasks to more junior members of a team, helping them and pushing them to grow and become confident in sovling problems.”

Senior Engineers who are able to delegate tasks free themselves up for more high level planning and work, paving the way for the next move in the career, maybe Staff and Team Lead level positions.

Adaptability

The Modern Data Stack and tech in general moves forward at a dizzy pace of constant change and innovation. It’s hard to be able to figure the marketing crazy from reality. Yet a Senior Data Engineer needs to be following what is happening in the industry, understanding what new tools and tech are truly game changing.

You can’t be stagnant and provide value for forward thinking Data Teams, trying to keep up on new releases, features being added to your current software, open source or otherwise, can save time, money, and headaches.

Kind, understanding, and empathic.

I can’t emphasize these character traits enough. You know what it really boils down too?

“You can’t lead people and processes very well if you aren’t simply a kind human being who is a joy to work with.”

Honestly, you have one life, you might as well be happy and have fun while doing it. You want your boss to be nice to you and promote you? You want more junior engineers to listen to you and follow you?

Here’s a novel idea. Be kind. Be pleasant. People inside and outside your Data Team will enjoy working with you, and that builds a sort of organizational capital that is worth something.

Make hard decisions.

Of course life in the Data World isn’t always rainbows, unicorns, and sunshine. Many data projects are riddled with emotion, difficulties, and money, all leading to high tension and lots of decisions that can go many directions.

Picking a direction and sticking with it, making the best decision possible is a skill that should not be under-estimated.

Too many decisions get stuck in unhelpful arguments, folks picking sides based on what they like or their favorite tools. A Senior Data Engineer can cut through all the unnessesary fluff surrounding issues and just make the best decisions possible at the moment with what information is available.

Admit Mistakes.

Some of the hardest words to even leave your mouth, ever since you were a kid up until this very momement.

“I made a mistake.”

There is something freeing about admitting you are a human, fraught with normal h human problems, like making a bad decisions. But, just because you made a bad decisions doesn’t mean things have to stay that way.

The best part about being open and honest is the ability to have people trust you more, and the option to fix the problem and move forward with a better solution! Everyone wins.

Breadth of knowledge.

I’m not nessesarily talking about depth of knowledge here, although that is a good thing. The more the depth the better. But, it’s extremely important for Senior Data Engineers to have a breadth of knowledge across a wide range of topics.

This is one of the skills that make a Senior a Senior.

Knowing a little about most cloud platforms (AWS, GCP, Azure), Big Data Processing tools, SQL and databases, SQL vs NoSQL, DevOps, bash, server basics, security and permissions, make someone a very valuable asset.

I know it sounds like a lot, but having the basics across a wide range of topics will help the Senior Data Engineer make better and more informed decisions.

Know what corners to cut.

Maybe cutting corners has a bad connotation. Is there a better way to communicate this idea? Knowing where to put more effort, and where to put less? There you have it. This is something that most all junior engineers get caught up in.

“Spending way too much time on un-important areas drags down productivity and has a major drag on a Data Team.”

Time isn’t infinite, you can’t just do what you want all the time. Projects have deadlines, some tasks are more important than others. In the end, being a productive solver of data problems requires that you understand where to put time, and where not to put time.

Aligned with the business and product.

A related topic is being in sync with business and product in your organization. Like it or not the Product and other business teams are critical to the success of the company you work for. They have valid view points and needs.

Ignoring the needs of teams outside of Data, or not prioritizing them above your own needs will have serious negative impact.

If you are not a team player it’s unlikely you will make to the Senior level, and have little chance of moving past that point. Being seen as someone who provides organizational value is a big deal for your career and your journey towards a Senior level roll and higher.

Works hard to write good, clean code.

I do want to emphasize that writing good clean code, and being at the top of your game is almost a prerequisite for becoming a Senior Data Engineer, or at least should be. Data Teams will look to you, as a Senior, to model best practices at all times and in all the work you do.

• Writing good unit tests.

• Writing DRY code.

• Keeping KISS at the front.

• OOP or Functional, doing it well.

• Knowing and keeping best practices.

• Not taking short-cuts.

• Taking the time to write readable, well documented, and maintainable code.

I’m not saying you have to be perfect or the top of your craft. But you should be on your way and that should be the trajectory of the projects and code you work on. Not everything is going to be perfect all the time. We make mistakes and take shortcuts when we shouldn’t. But, that isn’t who a Senior Data Engineer is.

In summary.

Of course I don’t think this list is all inclusive of what it takes to become a Senior Data Engineer, this is just my take based on my experience. There are all things I wish I knew in the beginning, of course I still would have had to learn them and work on them, but knowing this would have been helpful.

I was under the harmful impression in the beginning of my data life that all I needed to do was work on becoming the best at SQL and the best at programming. This just simply isn’t the case. In fact, if that’s all you do, it could be harmful to your long term career.

Learning to work with others and build no-technical skills is just as, if not more, important.