End of Year Engineering Planning for 2025
... how to do it ...
How many of you little hobbits are thinking more than 5 minutes from now? You’re probably just thinking about what new Snowflake or Databricks feature you want to play with. What to post about on Reddit. When should you finally call your mom to check in? That you should finally go to the gym. What you might get for Christmas.
Thanks to Delta for sponsoring this newsletter! I personally use Delta Lake on a daily basis, and I believe this technology represents the future of Data Engineering. Check out their website below.
The end of the year is nye upon us as the leaves change and the cold wind and rains start blowing in. The end of the year means one thing … you should have already started your Engineering planning for 2025.
Whether you’ve ever even thought about this process probably depends on where you are at in your career, or your position at work. But, even though ye may be a lowly sniveling weevil digging little holes in mountains of code, I dare you to stick your head up into the sunshine and do yourself a favor. Start planning, or learning how to do Engineering planning for the upcoming year.
If you are old or new, young or old, whatever, Engineering planning can be an esoteric sort of process that seems to be like staring into a crystal ball and saying strange incantations at midnight.
I think we can unravel that mystery for you. It’s not that bad, it’s just dreaming and thinking big, high level, and then bringing it all back down to earth.
Let’s jump into what that process looks like.
How to do Engineering Planning.
I’m going to make this as straightforward and concrete as possible, I really don’t think it needs to be that hard. Not doing Engineering Planning for the new year ahead is a grave mistake that too many Data Teams make.
It’s too easy to wander around in the dark lands of data, stubbing your toe and bumping your head against the rocks hidden on the wayside. You’ll never “get ahead” or “get on top” of things if you don’t plan.
Engineering Planning for the year ahead will …
Help you keep a clear ahead when you’re in the pits next year
Hold yourself and your team accountable
Enable you to grow stronger technically as a team and individual
Enable you to implement new features and technologies
Keep ahead of tech debt
Deliver value to the business
You are lost without a map, just wandering, doing a lot, and never getting anything done. The Engineering Plan, at a high level, will solve those issues.
This process outlined below is somewhat simplified, there is much nuance, but this is as good a place to start as any.
Mapping out the Engineering Planning Process.
Let’s break this process down a little more, from the above image, into actionable steps.
Business Priorities
The number one most important thing to do, even if it requires you kidnapping someone, is to get the C-suite/Executives to lay down on paper the Product Roadmap and their desires and priorities for the coming year.
This stuff will drive 80%+ of what Engineering works on in the next cycle.
The timing of the desired outcomes and products/features is key, we need to be able to lay them out over a calendar.
Engineering Wishlist
Second, after Business Priories, are the wants and needs of Engineering as a whole. What upgrades, tooling changes, infrastructure additions, or whatever that we “want” or “wish” to happen?
This is not tech debt, this is a wishlist of where “Engineering wants to drive towards in the future.”
This keeps people happy and engaged and the long-term viability of the Data Platform in play.
This will be a much smaller list than Business Priorities, probably 1-5 things might get picked from this list and done.
Engineering Technical Debt
It’s critical for the long-term success of any Data Platform that dealing Technical Debt is planned for, and put into the normal work cycle.
If you don’t PLAN for tackling some of your serious tech debt, your Platform is doomed to fail over time.
The rest will follow. It’s really about gathering all available plans and desires from Product first, then Engineering, laying them out. This gives a good idea of what the next year will generally look like.
It will force Product and C-Suite to communicate their plans and priorities (that you can hold them to).
It will give Engineering a general idea of what coming at them AND what they need to do to prepare for that work.
Engineering will have to prioritize their own wishlist and pick a few things from the list.
Engineering can hold themselves accountable for pushing the Data Platform forward.
It’s a give and take, you can’t have your cake and eat it too.
I view Engineering Planning for the new year as mostly an art, and less of a science. The thing is, you just need to do it. Some plan and some idea of what is coming next year is better than just winging it like most people.
Sure, you can just juggle stuff as it comes, work randomly on this and that when you have time, but to be a next level Data Engineering team you have to put expected actions and outcomes down on paper, you have to look yourself and the work in the mirror and make hard decisions.
It’s one thing to generally wave your hands in the air and say …
we want to adopt x, y, and z new tech stack.
we want to address this pain point.
it sounds like Product has some new things to work on.
It’s another to actually talk to real-life people, get stakeholder input, combine it with Engineering’s desired plans, and start making hard decisions and mapping out a year’s worth of work.
I mean it will probably only be 70% right if you’re lucky, but it will help you get the right things done in the right order.
Engineering planning can be summed up as making a giant list of TODOs, Must-Dos, Wish-Lists, and the like, and spending time thinking about …
How complex and long each project generally is.
What needs to be in what order.
Understand blockers and dependencies ahead of time
The ability to be forced to pick the few most important things that need to get done.
The ability to be held accountable to others and yourself.
Sharing and Verbalizing the Plan.
So once you’ve done the leg work, made a nice Google Drive doc, talked to Engineers, talked to Product, sat around staring at the trees and dreaming about your wishlist for Engineering … you have it all in one spot, the good, the bad, the ugly.
The next part is actually the hard one.
You must verbalize and disseminate this plan.
Putting a plan on paper forces people to be involved.
Expect pushback and changes, that is a GOOD thing.
The reality on paper with a planned timeline will always be different than what’s been in people’s minds.
So, be brave, be different. Make a plan. Write it down. Think about it. Talk to people about it. Make decisions. Decide what’s important, and what’s not.
How do you plan for Engineering in 2025?