Becoming a "better" Data Engineer
being more competitive
I get this question in one form or another, as regularly as my mother calling me on Mondays to check-in. It usually is formed something like …
How can I be better at my job?
How can I be more competitive (in this job market)?
How can I get a better job?
How can I move to a Senior+ level role?
How can I grow?
How can a person beat all the other people just like you running around out there? Maybe you’re looking for a new job. Perhaps you’re falling behind at work. You’re getting old and feeling left behind.
I get it. I’m old. Every day that goes by, I get another grey hair, and there are six new Engineers who are “smarter” than me. So how am I still hanging on by my fingernails when crossing the Big 40 mark? Good question.
Thanks to Delta for sponsoring this newsletter! I personally use Delta Lake on a daily basis, and I believe this technology represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.
I have no idea if this will be helpful, but I'll be sure to give you a 10-second overview of my background.
self-taught (no CS degree etc.)
worked at mostly startups (no FAANG or MAANG or whatever)
started working when SQL Server was your Data Warehouse
started as a Data Analyst
That proves anything is possible for anyone with a bit of elbow grease.
Also, I wanted to list many articles I’ve written on this topic generally.
Supercharge your Data Engineering Career.
Transitioning to Senior Engineer
When a good engineer goes bad.
Data Engineer vs Senior Data Engineer
What makes "smart" engineers so stupid.
My Journey as a Data Engineer - Visually
Maniacal Focus
New Year's Resolutions for Data Engineers.
What is a “Good” Data or Software Engineer?
What I’ve Learned After A Decade Of Data Engineering
The Best Piece of Software Engineering Advice
The Difficulties of Senior Engineer …. are not Engineering
4 Ways To Setup Your Data Engineering Game.
Before I offer any advice, we shall do the most classic thing and send all the above articles to ChatGPT and ask our new overload AI to summarize some of this career content for us.
AI - TL;DR
Across all twelve essays, the storyline is consistent: Exceptional data engineers differentiate themselves less by exotic tech and more by decisive action, relentless learning, disciplined focus, and human skills that scale their impact through others. Nail those, and titles, pay bumps, and leadership roles follow almost automatically.
Mindset Shift - code to code+
I've given some thought to how to summarize my advice to someone looking to grow to the next level, whether it's a new job, their first job, or simply a promotion. It isn't easy; everyone and every situation are different.
From my point of view there is no shortage of talking heads on every social platform under the sun talking about career advancement and how to nail your next interview. I’m so much interested in that topic, as much as how to build a long-term sustainable and well-rounded career.
All I can do is share what’s worked for me. I’m generally old school, so a lot of my career advice would be simply to “work hard.” In our fast-paced digital culture of ease and comfort, most people don’t work hard; if you decide to work hard at whatever you’re doing … you will automatically be “better” than most people.
Mostly, what it comes down to if you want to build a long-term sustainable career in Software and differentiate yourself from AI and the thousands of other (often better) programmers … is to stop worshipping at the feet of the Code God.
Yes, you should be an above-average programmer, but that has lost its luster in the face of AI that is cranking out code faster than you ever will. But you have to move beyond the code.
8 Universal Habits leading to the better you.
Decide, then own the outcome. Clear, data-backed calls beat committee paralysis.
Read, learn, repeat. Staff-level books, specs, and docs compound faster than tools.
Move before you turn into a fossil. New roles or responsibilities every ~2 years keep skills fresh.
Test & document your data pipelines. They’re the forcing functions of quality thinking.
Practice deliberate kindness. Empathy turns raw IQ into team-scale impact.
Broadcast what you know. Blogs, brown-bags, and code reviews turn you into a multiplier.
Guard your calendar. “Maniacal focus” on the highest-leverage task beats 10 busy days.
Stay humble. The moment you must be the genius, you’ve stopped learning.
Yes you need to take the time to become a technical savvy Engineer, but this probably includes reading more documentation, POC’ing new tools and technology, and working on high level architecture and design.
Your second focus should be all the NON-technical things we just got done talking about. Yeah, there is a lot of them, but also, the climb to the top isn’t easy either.
“Be good at software—be better at everything around it.”
“Focus is saying no 90 % of the time.”
“You don’t need to be the smartest person in the room; you need to make the room smarter.”
Can you shortcut this path? Not really. But you can start today at whatever job or position you are in. Typically, extra effort will be noticed eventually, if not, move on, don’t waste your time.
Work hard at your craft, love what you do, don’t get comfortable, always be pushing forward to learn more both technically, and non-technically.
