Most Data Engineers are Mid

Stuck in the doldrums? Read on.

Well, you might not think it’s a very nice thing to say. But that doesn’t mean it isn’t true. I’ve worked as a Data Engineer for a long time, worked with many of them, and interviewed many of them, and I think my sample size is just right.

Most all the Data Engineers that I met are mid, at best.

It’s actually not hard to stand out these days. I’ve experienced endless piles of electronic resumes washing over me with boredom and single-minded sterileness, like the endless crashing of the ocean on some desolate island shore.

I’ve stared in wonder at many bulbous and deformed creations like some Frankenstein electrified in a gloomy dungeon, code, SQL, and ideas that have spewed forth, bubbling rancid cauldrons that devour the souls of Data persons unhappy enough to step into them.

All the while, their creators, smile and grin like some youth devoid of understanding, completely unaware of the monsters their own hands have made, that are about to turn and tear them apart.

Thanks to Delta for sponsoring this newsletter! I personally use Delta Lake on a daily basis, and I believe this technology represents the future of Data Engineering. Check out their website below.

No one wants to be mid. At least I don’t think so. Is it my old age getting to me? Maybe I’m becoming the old grumpy person in the neighborhood, sitting on my digital porch of sorts, bemoaning the state of things.

What does a mid Data Engineer look like?

Mid Data Engineers are easy to spot. Why? Because they are everywhere, all around us. The reality is you will rarely meet an Engineer who doesn’t think they are a gift to humankind. There must be something in the Data water we drink, so strange sickness that takes hold of people. That seems to be the only explanation for the heinous crimes I see perpetrated against our precious Data.

I’ve always caught that path that is closest to the edge, against the grain. This rambling will be no different, I will call out what I see, and exhort others to abandon their evil ways and come to the light.

Contrary to popular belief, a mid Data Engineer is very easy to spot in the wild. You don’t even need your binoculars. I can sum that mid Data Engineer up with a few bullet points.

• Never Act

• Poor Quality

• Never Learn

• Lone Wolf

So what about those of us who don’t want to be mid? Well, we can just train ourselves in the reverse. Let’s go through each of them.

Never Act

The mid Data Engineer never acts. What does this mean? It means constant hand-holding, never-ending questions, and the ability to NEVER make a decision or just try something. Stagnant. Sitting. Never moving.

There is a difference between careful planning and not doing anything. There is a difference between thinking about a problem and being unable to move forward. Code is dirty, Data is dirty. Working in data is a master class in flexibility and divining the opaque.

You can’t rely on a mid Data Engineering to …

• Just make a decision.

• Use their past experience to inform the decision today.

• To be a self-starter.

• To plan a project.

• Just write the code.

• Unblock themselves.

• See the big picture.

I could go on. Right or wrong, as an Engineer who wants to grow, you’re going to make mistakes. But the only way you will ever make a mistake is if you are actually doing something.

Failure is learning. Getting yourself through hard spots, even when it’s frustrating builds the muscles that allow you to move and act in Senior level roles.

It might seem like a small thing, but having an engineer who can deal with ambiguity, make decisions, and just get something done without a lot of fanfare … well, that’s worth something.

Poor Quality

What’s the next worst thing after a mid Data Engineer who can’t seem to make the decision to write code? How about poor quality at every level?

Honestly, it doesn’t have to be code either. It’s everything. Poor Quality …

• Documentation

• Design and Architecture

• Code

• Testing

• Communication

• PRs

When it comes to Data Engineering poor quality is a killer. It creates more problems than it solves. Better to not do it, rather than spew out streams of half-done code that either doesn’t work or only sometimes works. Why? It creates frustration, bugs, and on-call problems, and increases exponentially the difficulty of future development.

Poor quality is hard to deal with. Mostly because I suppose we should be happy “something is getting done.” Yes and no. Of course, there are times when we have to move fast and get things done. But, there is a line drawn in the sand somewhere.

Poor Quality is a habit, it’s a way of life.

It could be simply someone new to a field, maybe, or it could be laziness or simply not caring, who knows? The worst part of Poor Quality? Many times it’s context-based. It depends on the work environment and expectations around you.

One person’s crappy code could be the best thing ever in another context! What does it boil down to? Is the work you are doing lifting the ship or sinking it?

Never Learn

This one isn’t that hard to guess at. A mid Data Engineer will not learn. Happy to sit like a frog, stewing in some mucky dark pond. It’s probably the worst place to be, it’s a hard pit to dig yourself out of.

I’ve heard endless complaints about not having enough time, wanting to have a life, don’t work so hard, etc. Sure, I get all that.

Spending your weekends on Leetcode is not what I’m talking about. I’m talking about an attitude of learning that is natural to your way of life.

Learning means you care. Care enough to simply spend a minimal amount of time over your entire life simply trying to improve yourself by various means in your vocation of choice.

• Read a technical book.

• Read blogs.

• Read the documentation of tools.

• Be interested in new things.

• Pay attention to the macro-environment around you.

• Teach yourself something new here and there.

Notice I didn’t say to read 15 books a month, contribute to open-source software, write your own blog, read 15 others, and the list goes on.

No, I’m talking about simply an approach to life in which you want to keep up and better yourself.

If you can’t find it inside yourself to simply read a book or two a year, read a few blogs, or try something new … then I can’t help you, nor can anyone else.

Mid data engineers don’t take learning seriously. Things are what they are, they settle in for the long term.

Lone Wolf

Last but not least, the Lone Wolf. Well, I have to be honest with you. I love me a good Lone Wolf. I have a perpetual desire to be a Lone Wolf. I prefer the Lone Wolf.

But, in the world of the eternal Grug … Lone Wolf bad.

The problem with the Lone Wolf mid Data Engineer is that team cohesion and the efficiencies that come along with that … are impossible to foster and generate.

• mid Data Engineers work by themselves, for themselves.

• Lone Wolfs are bad for long-term team happiness.

• Lone Wolfs can hog knowledge that needs to be shared.

I’m not sure what else to say about Lone Wolfs. I love them and hate them. Because I find myself slipping into Lone Wolf status myself.

But don’t call me a mid Data Engineer. I will find you.