New Year's Resolutions for Data Engineers.
Start the new year off with a bang.
2023 is your year, it's the year of the Data Engineer. Let it be one for the record books, a pivotal change for yourself. Can you teach an old dog new tricks? I have my doubts, b they say hope springs eternal.
Maybe you're at the top of your game, or at the bottom. Either way setting 2023 New Year's resolutions for your Data Engineering life is a sure way to transform your digital self into the chiseled and fined tuned behemoth.
Let me give you 5 suggestions you can set as resolutions to improve your life.
I will unit-test all my code.
I will study the art of Data Modeling.
I will learn one new programming language.
I will read at least 2 new technical books in 2023.
Well, that should keep you busy this year eh?
I’ve found at different times in my Data Engineering I’m growing faster, than sometimes slower, and it usually has to do with me. Yes, life gets busy, and work gets crazy, but in the end, that’s the way it always is.
Finding your way through the crazy, being intentional about your schedule, and having focus time to learn and grow are essential for your career and overall happiness.
I believe that growth and learning require sacrifice, but not as much as you think, it can be enjoyable and fun, paying dividends back to you for years. Let’s just jump into my 4 suggestions for New Year’s Resolutions, Data Engineering style.
Unit Testing
Your first task in the New Year is a big one, and it’s called unit testing my friend. Whatever the reason is, Data Engineering has always lagged behind in certain Software Engineering best practices, and the number one item is unit testing.
Why unit testing? Because it’s the gateway to a better future, that’s why. It’s always the first step for every Data Team when working towards a better way to develop and do Data Engineering.
When you start down the road of writing unit tests for data pipelines, it starts to require other changes to your Data Stack, and processes that make can totally change the trajectory of Data Engineering.
Unit tests will drive you toward functional code ... that tests easier.
When you unit test your code, you will write better code.
Unit-tested code is easier to reason about and debug.
If you’re already writing unit tests, then I’m preaching to the choir. If you’re not, it might sound like a small simple step, but it’s a true gamer change. A Data Team that unit tests all data pipeline transformations and code will be more forward-thinking, have fewer bugs and generally be more efficient.
Data Modeling
There have been a few hot topics in the DE community over the past year, and data modeling keeps coming up as one of those topics and will continue to do so for some time. Why data modeling?
“With the rise of Big Data and the popularity of Databricks, Snowflake, Redshift, and other storage systems like Delta Lake, LafeFS, and the like, it’s become clear that no Data Platform is immune to a bad data model.”
It’s hard to find good resources to read up on data modeling, especially data modeling for Big Data applications. I would suggest if you are new to the Data Warehouse and Modeling landscape, to start with the Data Warehouse Toolkit, although it may be a little dated, most of what it teaches is foundational and timeless.
Do yourself a favor in 2023, and gobble up as much data modeling material as you can.
Learn a New Language
It’s a New Year, so we need to add some more new things to our repertoire of skills, a new bunny to pull from our hat so to say. I have found that there are very few things in my Data Engineering career to propel my skills forward than learning a new language.
“There is something other-wordly and exciting about it, starting from scratch, going through the struggles all over again. It keeps one humble.”
In 2022 I tried how not one, but two new languages, Golang and Rust, which I’ve written about a few times. I can say for sure that it made me a better Data Engineer. Learning new languages forces a few things …
Gaining a greater perspective of options when it comes to languages.
Remembering the struggle of learning something new (humility).
Learning how to approach a problem differently.
More tools in your box for future problem-solving and solutions.
You become a better programmer at what you’re already writing.
Learning new stuff keeps you from stagnating, turning into a stinky pile of rubbish the younger generation turns their nose up at. I suggest starting with Rust online book or Golang.
2 New Technical Books
You remember in school when you just played around in study hall when you should have been studying? Is that still you today? Time to change your wicked ways.
There are few things better for the soul than reading, it’s a great way to escape into another world, and learn something why you’re at it. Sure I know you can just read all the stuff you want for free online.
But, few things go in-depth like reading a technical book that someone has poured their life into. It’s almost impossible not to soak up some serious knowledge when reading such content.
Setting some time to read two new books in 2023 will be good for a few reasons. It forces you to set aside some time for growing and learning, it’s a good habit to develop. Also, it’s your chance to learn something about a topic you’ve been wanting to in the back of your mind, but for whatever reason haven’t gotten around to.
Here are my recommendations in no particular order.
The Pragmatic Programmer - basically about the theology of programming.
The Staff Engineer’s Path - What it means, and how to become a Staff Engineer.
Staff Engineer - interviews and talks about how to expect and how to become a force multiplier.
Fundamentals of Data Engineering - general info.
Spark: The Definitive Guide - become a Spark expert.
That’s just a few I’ve read lately, I’m sure you can find others that suit you more if you poke around.
Closing Thoughts
I really hope 2023 dawns bright and new for you, it’s been kind of a downer end of the year this year in tech, layoffs, and the like. But, that’s life and we have to look forward with optimism to this new year.
We can’t control many of the crazy things happening around us, and to use. But, there is hope, we can control ourselves and how we spend our time.
I think we can all agree if it’s not one of the topics we just covered, it might be something else, there are always areas we know we need to grow in to become the best and most productive Data Engineers possible.
I will unit-test all my code.
I will study the art of Data Modeling.
I will learn one new programming language.
I will read at least 2 new technical books in 2023.
Pick one from the list and get started! Don’t forget to reach out and let me know how it goes!!