Well, I had my moment, the tipping or boiling point, I guess you would call it. I’ve had enough, and, in a moment of fury, I ripped Polars from its Lambda throne and supplanted it with DuckDB. Been a long time coming.

Look, before you feed me to the wolves, you need to get the whole picture. Don’t burn me at the stake or stick me in the dungeons of Castle de’cliff without a trial.

Let me spin you a tale, and you be the judge.

Ok, turn back the dials of time to circa 2022-2023. I’m usually an early adopter of technologies, and I pride myself on kicking the tires on new and shiny tools, taking people to task for what they say a “thing” does.

The proof is in the pudding, and one of the most popular articles I’ve written on my other blog was how to replace Pandas with Polars back in the Year of Our Lord 2023.

• Heck, I put Polars in production workloads in those early days, when others were still watching from the sidelines.

Yeah, it had its rough edges from the beginning, but the speed of Polar’s Rust-based design was out of this world, and it was the tool I’d been waiting for a long time. The ability to work on large datasets in a streaming manner, replacing Spark in many workloads, was a dream come true.

Come on, bro, this is just a small sampling of all the content I’ve written on Polars over the years, just a subset.

What I’m trying to say is that I’ve used Polars for years in every possible situation. You know what I’ve learned over all that time?

At the end of the day, we want tools, wait … we NEED tools that are reliable, stable, where the maintainers take ownership to another level, and build with developers and their main use cases seriously.

Hey, I’m just a guy with an opinion; you can have yours. I’ve been fighting Polars issues in production for years now. Usually, I’m willing to just figure out a fix and move on, roll my eyes, etc.

I first got a bad taste in my mouth when I ran into memory issues, found the same problem on Polars' GitHub, only to have someone who wasn’t very nice close the issue as “not our problem.”

What made me switch from Polars to DuckDB?

I recently rebuilt some AWS Lambdas whose main function was to use Polars to read some data from S3, transform it, and write it back to S3. Ya’ know, I’m not a Luddite, pinned Python versions using the base AWS Python image.

Something like …

FROM public.ecr.aws/lambda/python:3.13
COPY .  ./
RUN pip3 install polars==1.31.0
....

I was only updating some logic, nothing else. Little did I know I was in for another Polars Easter egg. Of course, I woke up the next day to find Lambdas not working.

Should have been more rigid in my testing, I suppose. But we get complacent, and that’s how it usually works. Fix or upgrade some logic, something totally unrelated goes pop.

Yeah, some might argue that this sort of thing is probably a consequence of dual evils, Python environments (packages tied to packages tied to packages… supply chain), and life in Software.

In the perfect world, with the perfect set of tools and unlimited time to ponder the comings and goings of the various software engineering best practices, such things can be caught early and dealt with.

But, I find myself stuck in reality, competing priorities, limited resources, and time; you do your best within the constraints given. It was the straw that broke the proverbial camel's back. I, with only a slight tingling of remorse and with prejudice, ripped Polars from production and replaced it with DuckDB.

I sleep better already.

Read more

When you first consider a data platform as a whole, the architecture and infrastructure serve as the foundation upon which everything else is built. For our purposes, we can use the two words, architecture and infrastructure, interchangeably, not precisely, but close enough.

The true cornerstone upon which you will build the entire house, the canvas on which the data picture is painted.

Without a well-defined architecture (logical) and a strong infrastructure (physical) to support a system, the rest of the platform will struggle to function efficiently, and execution will be hindered.

The biggest challenge in designing a data platform is that no two architectures (logical) look identical. Different business requirements, technological stacks, and industry constraints influence how a data platform is built.

However, common architectural patterns and best practices can guide the development of a resilient and scalable system. Yet, as I’ve learned over the decades working in and building data platforms, it’s more than simply picking the “right tool.”

Take, for example, three made-up companies for this discussion.

• Acme Mfg Corp.

• Acme FinTech.

• Acme AgTech.

The businesses themselves could not be any different: the manufacturing of physical goods, the movement and transactions of money, and the technology side of “Big Agriculture.”

It would be no surprise that, from a technology perspective, the data platforms supporting these different operations would look completely different. But, surprisingly to some, the fundamental concepts underlying the “how” of building each of these data platforms would be the same.

It doesn’t matter whether we are producing widgets on an assembly line or processing satellite imagery of a farmer’s field; both data platforms need, for example, monitoring and observability in place.

Today’s post is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.

Define

Back to the task at hand, before we get ahead of ourselves, we should define the world architecture in the context of data platforms for our discussion, so we are on the same page.

How about infrastructure?

Ok, now that we have those more educational definitions out of the way, what does that mean when the rubber hits the road?

The architecture of a data platform is where you put ideas on paper to describe and visualize how you want to approach building and bring together the components that make up the system.

The infrastructure is where we start building the technical details and tools we will use to implement the architecture and bring it to life, and make the decisions on which tools and frameworks to use.

• Architecture is one of the most challenging transitions for an engineer. They move from a tactical role, such as writing individual data pipelines, to a more strategic role involving platform-wide decision-making. Instead of focusing on a single database or a specific ETL pipeline, platform architects must consider how the entire ecosystem fits together, ensuring that every component is built for performance, cost, and maintainability.

It’s also clear that we’ve moved from a time when building data platforms was solely the job of an architect to one where even Senior Engineers are expected to develop and maintain them, many times, from scratch.

This can be a hard transition, from the minute details of a daily pipeline to thinking about the entire systems upon which that pipeline operates.

Subscribe now

Core Architectural Principles of a Modern Data Platform

Before diving into infrastructure and implementation details, let’s explore some guiding principles that should shape every decision when architecting a data platform from scratch, or trying to modernize an existing system:

• Scalability: The platform should handle increased data volumes and additional workloads without significant performance degradation. It should scale.

• Resilience & Fault Tolerance: Systems must be designed with failure in mind. Failures should not cause significant disruptions, and data should not be lost due to single points of failure. Can the data pipelines be rerun with a single click, or without any clicks at all?

• Modularity & Flexibility: Components should be loosely coupled so that individual parts of the system can be upgraded or replaced without significant disruptions. We don’t want to over-tighten couplings between Platform components.

• Security & Compliance: Data governance, access controls, and compliance with regulations such as GDPR and HIPAA must be incorporated into the design from the beginning. Who can do what, with what?

• Observability & Monitoring: Engineers must be able to detect failures, measure performance, and diagnose issues effectively. Insight into system components is key.

• Cost Efficiency: Designing the platform with cost efficiency in mind ensures that the business derives maximum value from its infrastructure investments. Don’t overengineer or complicate the architecture.

Also, while discussing the above technical approaches to bringing a data platform to life, we can’t forget the real-world value and intended purpose of the engineering systems we will build. It’s extremely easy for engineers at all levels to get overly excited about technical details, and that is a good thing, but it cannot be done in a vacuum.

• There is another significant piece of architecture that is both the bane of most engineers and also the most important. The business requirements. Don’t you just love those words?

It seems this is where most engineers start to have their eyes glaze over. They just want to solve technical problems and stay as far away from the boring business stuff as possible. I have some bad news for you.

If you ignore the basic business requirements related to a data platform, you are building a house of cards on sand that will be all for nought, a waste of your time.

Share

Applied examples.

Think about how different the business requirements for our three companies, Acme Mfg Corp, Acme FinTech, and Acme AgTech, are simply from the nature of their operations. If we were to sit with the C-suite from each of these companies, as a technology consultant, and were to ask the CEO and CTO what their core principles are, they might say …

• Acme Mfg Corp.

  • We want to deliver high-quality products while reducing costs to our end customers through our wide distribution channels.

• Acme FinTech

  • We want to provide cutting-edge payment solutions with high availability, performance, and zero downtime.

• Acme AgTech

  • We want to provide an integrated solutions platform that delivers insights at scale to help farm operations drive profitability and sustainability.

Without knowing anything else about the operations of these businesses, an astute observer can already sense differences in each business model that will directly impact the data platform, and how one should build and maintain one; words like cost, performance, zero downtime, and scalability have a real-world impact on what a data platform underpinning these businesses would look like.

Business Value and Purpose

Under no circumstances should anyone start the architecture and planning process for either a greenfield (new) project or an upgrade/migration to a data platform without talking to and understanding the business at its core.

When I say business, I mean every single non-engineering or technical group that might have some sort of need or impact from the data and insights being produced. To be very clear, this is every single group or department inside a company.

Just as we saw above in the example of our made-up companies, the simple 
one-sentence purpose and intent of a business impact the “how” and “what” 
of a data platform, let alone the deeper, day-to-day operations in these 
different environments.

Here is a list of questions (non-inclusive) you could (and should) ask of your non-engineering counterparts and teams. Their answers will directly impact how you approach the data platform architecture and technical implementation.

What kind of data do you use or want access to to do your job better?

What data or data insights impact our end customer/user?

What data or data insights help this company provide better results/meet goals?

How often do you need the insights (freshness) of data?

How do you interact with the data today, and how do you want to do that in the future?

What is the budget for the data platform?

How big is the data team, and is it growing or stagnating?

What do you wish we could do with data tomorrow that we can’t today?

The hard part of aligning the business’s needs with engineering’s desires is that you have to read between the lines. The company (besides engineering) isn’t technical and often doesn’t even know what’s possible.

Yet at the same time, the other departments, like Product, Marketing, and the C-Suite, are consumers of the data and insights we produce. Meeting their desires and expectations is key to building a healthy data platform and a Data Team that is seen as an integral part of the business. It’s also not surprising to have many teams and people with competing desires and data needs.

What might this look like in the real world? How could answers to some questions and the desires of different business units inform our data platform architecture?

• Do we need batch, real-time, or near-realtime insights?

• Does the business trust the data it has?

• Does the business see data as an asset or a cost center?

• What kind of analytics tools are expected, and what features are needed?

• Do you have or need better Machine Learning capabilities?

• What is the budget for the data platform?

• What new products are coming?

• Can we support those new needs on the current platform?

These are the real-world answers that will have an outsized impact on how you move forward with building a data platform. Do you need a Kappa or Lambda approach?

Will there be heavy use of data visualizations and analytical needs? Does the business need and want to interact with the data, or just consume basic reports? What are the expectations around the freshness of data? Are we going to have to be extremely budget-conscious, or is that not an issue?

• All of these decisions have real-world infrastructure-related impacts; it might be the difference between a more costly SaaS version of some software, vs a cheaper but more time-intensive self-hosted solution.

What if we take just one of these business questions and apply it to our three companies?

• Do we need batch, real-time, or near-realtime insights?

  • Acme Mfg Corp.

    • Medium-sized widget-producing factory that sends large orders to various distributors, which is very cost-conscious; batch systems of dealing with data insights are more than enough to meet business needs.

  • Acme FinTech

    • A high-tech company that deals with financial transactions; the performance and criticality of payment systems require streaming insights to meet business needs.

  • Acme AgTech

    • Medium-sized tech company working in the Big Ag space, dealing with scale; they have an interesting mix of needs, a large customer base, and huge datasets. Near-realtime will provide sufficient insights.

Only by truly understanding how a business operates can we find the answers to how data is produced, processed, and consumed to provide value.

It can be easy to gravitate towards what we are most comfortable with and ignore those topics we deem less necessary. This can be a very costly mistake.

Once a data platform is up and running, it’s much more challenging to make significant changes without disruption, like changing a flat tire while the car is still moving.

In the upcoming parts of this series, we will dive a little deeper into each item we listed above, which are essential to consider when understanding or designing the architecture and infrastructure of a data platform.

In Summary

Tools and frameworks change, and what is popular changes over time. Data platforms should be built and designed around a set of concepts and principles that rarely change.

• This is the danger when we discuss technical topics like building data platforms. With the fast pace of technology changes, it’s essential to talk about timeless truths that transcend any single technology or concept(s) that is closely tied to them.

Architectural decisions define the success or failure of a data platform. While individual tools and technologies evolve, the foundational principles of scalability, modularity, observability, and cost-efficiency remain constant.

Come follow me on YouTube!!

Look, you don’t have to, but if you want to, here we go. Also, you should probably go find one of those elusive AI Engineers that are now showing up on LinkedIn full of wisdom and foresight. Maybe they know what’s going on. Either way, I’ve built a few multi—agent POCs, some on the Databricks platform and others more bespoke.

I’ve poked and prodded enough “AI stuffy stuff” to at least make myself dangerous. Put me in, coach.

I’m just a dude on the internet who’s noticed the uptick in jobs involving “AI” or “Gen AI,” as the hipsters call it. And, heck, why not throw my hat in the ring? Everyone is already freaking out about losing their jobs to AI.

• The inter-webs are awash with news of Corporate Overlords slashing jobs in the name of AI efficiencies. Real, or no? Well, that is still hotly debated.

Today’s post is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best in class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems – current and future – are synchronized around the same datasets, updating in milliseconds.

AI Hype

Even those filthy rich stinking bankers are cashing in on the hype of AI, putting thousands of jobs on the chopping block of AI.

Heck, even our old friend Zuck seems to be getting swept up into the craze, like a lemming; they must all jump from the same cliff. Once one goes, you know the rest will follow soon. Meta is reporting record profits, but that ain’t stopping them, no sir Sunny Jim, once a hole in the ye ole’ dam has been poked, the waters start to flow quickly.

Well, where does that leave peons like us? Fighting for scraps in the post-apocalyptic future, most likely.

Either way, I prefer to find the bright spots if I can. They are out there if you know where to look. A little pragmatic attitude, half-glass-full sort of stuff goes a long way in overcoming obstacles of all sorts.

I guess if we can’t beat em’, we must join em’.

I’m not talking about simply using AI as part of your software development workflow; I’m talking about building AI and Agentic systems. Pull back the curtains and see what Oz is up to.

I certainly have my ideas about what a Data Engineer must learn to become an AI Engineer, but instead of guessing, let’s head over to LinkedIn and run an informal search for the skills and tech companies that are asking for in “AI Engineer” job descriptions.

At the end of the day, if you want to be a Gen AI Engineer, then we have to have and learn the skills that are being hired for. Let’s do this like proper engineers.

Old school. Heck, we might even write this code ourselves, *gasp.

I scraped 30 AI Engineer job listings from LinkedIn into text files. What we can do is just do a simple Word Count, maybe with a Word Cloud, and actually see which skills are required for this job.

All this code, including the text data for the jobs, is here on GitHub. Enjoy.

The Code.

Here is a sample of one of the text files.

So, now let’s write some Rust to do the dirty work, find out what in the world all these AI Engineer jobs are all about.

Now, I found the results very surprising and funny, not what I was expecting. I mean, indeed, there were things in there like RAG and Agents, but I was assuming I would see words like … inference, langchain, langgraph, api, frontend, etc, etc.

Indeed, it was the opposite.

In a good way, I’ve seen firsthand that building AI systems is more about systems design and architecture, with a few specific tools thrown into the mix.

cargo run -- word-count

And the word cloud …

cargo run -- word-cloud

I’m not sure what this says more about: the AI landscape as a whole, or companies’ take on what makes a good AI Engineer.

Again, as someone who has built my fair share of Agentic systems in Databricks and elsewhere, I pretty much agree with the requested skills list. It’s easy to pick up a framework, like LangChain; it’s not easy, without experience, to have Systems Design experience that assists with building multi-agent workflows … because that is what you are dealing with … multiple systems tied together.

Let’s dive into our findings and summarize what is needed to move to an AI Engineer position.

Top 5 Skills for AI Engineer

I want this to be practical from my experience over the last few years building AI systems, as well as what the actual data says, pun intended. Good intentions are just that, but seeing this word analysis of 30 job descriptions for AI Engineers helps solidify that we are going down the right path.

(To be honest, it just sounds like a good data engineer, which, from my experience, is true, besides a sprinkle of odd toolsets.

1. Understand Systems Design (architecture)

When I first started playing around with Agentic and AI systems, building them, that is, I noticed something I was not expecting. Basically, I was building a bunch of separate “systems” that needed to talk to one another and work together.

• It was a classic systems design problem.

We see this playing out in the fact that the most common word in AI Engineer job descriptions is “systems.” Why? Because the business use case for LLMs and Agents in real life often requires data and integration across a suite of cloud resources, endpoints, and tools.

        [ User / App ]
               |
               v
        +----------------+
        |   AI Agent     |
        | (Reasoning)    |
        +----------------+
           /    |     \
          /     |      \
         v      v       v

 [Vector DB] [API Calls] [Data Lake]
  (Memory)   (Actions)   (Raw Data)

         \      |      /
          \     |     /
           v    v    v

        +----------------+
        | Orchestration  |
        |  (Glue Logic)  |
        +----------------+
               |
               v
        [ Cloud Services ]
   (Compute, Auth, Storage)

Each is a separate system that must work together to complete a task. If you are curious what this might look like, read my article, Agentic AI for Dummies, or maybe my article Building Agentic AI … Fancy.

Clearly, to make the leap to AI or Gen AI Engineer, you must be adept and comfortable with Systems Design, something that has been around for a long time.

• If you want to learn more, check out this GitHub from on Systems Design.

Talking to databases, APIs, endpoints, tools, MCP servers, vectors, other agents, the list goes on. Add in the complexities of logs, traceability, monitoring, performance … across these complex workflows and apps that do all of the above … those are classic system design problems.

Share

2. Understand Multi-Modal Data (and generally data)

I had to laugh a little when I saw it, but this is also something I’ve commented about a number of times. Building LLM and AI systems is 90% data pipelines, sprinkled with a little orchestration. Data shows up as the second most common word.

Most* AI and Agentic systems I’ve dealt with are very heavily reliant on data. Could be Lake House data in Delta Lake or Iceberg, Postgres tables, RAG-backed non-tabular data like PDFs, etc. Either way, these AI Agents typically require substantial context and interaction with data, vector- or otherwise, to be considered noteworthy.

We live in a data-driven world; it’s been this way for a long time, and the AI boom has only intensified. No surprise that building AI systems requires an expert-level understanding of data.

Here is where classic Data Engineering diverges from what is needed for AI systems. Yeah, we can hook up that tabular data in Postgres for the Lake House via numerous connectors, but a large portion of the data needed for integration into LLM systems isn't tabular.

• RAG and Vectors

• Knowledge Graphs

For example, read the article below where I built a Databricks Knowledge Assistant on top of PDFs and Word Docs. This is not normal data engineering.

Most of use are not used to dealing with messy and unstructured data like this. You can’t just throw it all in a S3 bucket, I mean maybe you could, but long term and scalability wise, not a good idea.

With the rise of Agentic workflows, we are no longer dealing with classical data models where tabular data rules. Well, yeah, we are, but we are also now dealing with audio, video, PDFs, … just a large swath of data we historically haven’t been integrating into our data platforms.

For example, this is an excerpt from an AI Engineering job description.

Data Engineering Excellence

- Strong experience normalizing data from disparate sources with conflicting formats
- Obsession with accuracy where 99% is insufficient—compatibility data must be correct
- Experience building automated validation and conflict resolution systems
- Ability to model complex business domains (you'll learn heavy equipment specifics)

… or from another one …

Experience with cloud infra for AI, data pipelines, orchestration

At the end of the day, AI Engineers appear to be Systems architects, specializing in data and data movement across these systems.

If you’re curious about strangely shaped data, hard problems, and what the future might look like, read this article from Byoyant Data.

Share

3. The Cloud is King

This one I was kinda surprised at, not sure why, it sorta makes sense, but at the same time it seems like table stakes. Every once in awhile I run into a poor soul who hasn’t had the pleasure of being smashed upon the rocks of AWS and IAM.

Usually, it’s some corporate thing, where “the cloud” is kept under lock and key, and far away from the prying fingers of a thousand mildly bored devs.

I can feel that CTO’s axiety.

To be honest, I’m not going to spend a lot of time here, other than to say that, yeah, anyone working on AI systems is probably going to be working at a “forward-thinking company” that relies heavily on cloud infrastructure for all operations.

My personal opinion is that “cloud” in this context has something to do with …

• CI/CD

• IaC

• Saas Offerings

Here are some excerpts from the AI Engineering job descriptions …

- Cloud databases and production deployments.
- Comfortable with Docker, cloud (AWS/GCP), fast iteration
- Cloud and DevOps experience
- Cloud environments such as AWS

It’s just them saying what we already know: if you’re going to be an AI Engineer, you’re going to be working in a lot of the public clouds (AWS, GCP, Azure), etc. As well, using many SaaS provided services, maybe for inference, that exist in someone else’s cloud.

• These are AI systems that are being designed and deployed; they have to run and be deployed somewhere, that somewhere is not under someone’s desk, but rather with CI/CD in cloud-based environments.

It’s honestly just a good reminder that if you still don’t have much experience working with and deploying to major cloud providers like AWS, you should probably set up an account and start playing around.

Subscribe now

4. Workflows

Not going to lie, when I saw this word show up at the top of the list, I might have let a little squeal escape my lips. It felt like validation, a lifetime of Data Engineering that actually turns out to be useful.

Also, validation about what I’ve felt about building Agentic systems. Building these AI systems is an exercise in building pipelines and workflows.

For example, read this article on some of the challenges of building multi-agent systems.

At the core, we are building workflows between data systems and agents. Agents use tools in workflows; user interaction happens through workflows; and the data and logic used are wrapped in workflows.

If someone has never built workflows, especially large data workflows, it would be difficult to be dropped into the title of AI Engineer, knowing that a large part of the job would be building workflows to support the AI systems and Agents.

The difference between good and bad agentic systems and their workflows is nuanced, and that knowledge comes from experience.

The truth is, as well, with the rise of Agents, not only for code generation but also as entities taking actions and completing workflows themselves.

A person should be familiar with workflows if they are going to build Agents and AI tools to complete or participate in them. How do you monitor, observe, and measure the outputs or results of those black box agentic workflows?

• An example of this would be the following article and use case (a real use case) where an Agent (llm) IS the workflow and pipeline.

This is a new sort of frontier with a lot of exploration; everyone and their mother is building Agents and letting them loose to “do things,” which usually means completing a set of tasks, also known as a workflow.

This raises all sorts of interesting questions, like how do you build “deterministic” AI systems?

And so, we can all agree that “workflows” indeed make sense as one of the top skills required for an AI Engineer. Onwards and forward.

Share

5. Frameworks and Python

I sort of cheated and wrapped these last two skills into a single one, because they are basically the same thing. Python and Frameworks basically tied.

Sure, all the Rust hobbits will turn their noses up, but that’s nothing new. Python has been running the Machine Learning world long before LLMs were a thing, and will continue to do so for the foreseeable future.

Why “frameworks” and “Python?”

Well, the AI world is still young, but you can already see solidification in which frameworks are most often used. I’m sure things will rise and fall in popularity, but if you want to be an AI Engineer, you need to be familiar with the people frameworks (mostly Python) used to build these AI systems.

Here are some excerpts from the job postings.

- NLP/LLM Tools: Transformers, LangChain, OpenAI APIs
- Extend and improve our agent framework (built on pydantic-ai) and 
the infrastructure that supports it
- ML/AI Frameworks: scikit-learn, TensorFlow, PyTorch, LangChain, OpenAI API integrations
- Proficiency in programming languages such as Python and familiarity 
with AI frameworks (e.g., TensorFlow, PyTorch).
- Understanding of Agentic AI frameworks and tools, like CrewAI and Azure AI Foundry.

And the list goes on.

This is really no surprise at all. If you were a front-end engineer, they would ask you for the latest and hottest JavaScript framework, or whatever.

If you were a Rust engineer, they would expect you to be an expert with Cargo, or maybe Tokio for async operations. If you want to be an AI Engineer, then you need to use and be familiar with the tools and frameworks commonly used.

• Most of my experience lies with LangChain and LangGraph, both of which I highly recommend. They are great places to start.

So what are you waiting for?

Well, I found that interesting, even if you didn’t, you little complainer. With all the AI hype + doom and gloom, it’s easy to get caught up in the world-is-ending hype train. What can I say, I’m a sucker for it too.

Either way, I guess we can all use the escape hatch of transitioning to “AI Engineer,” if all else fails.

Now that we have pulled the curtains back on that elusive job and seen what skills are required, it’s a little less scary. Maybe.

Methinks, probably because I’m biased, that Data Engineers are well-suited for these roles of AI Engineer. Working with data, workflows, frameworks, systems … this has been our life for a long time.

I do not know what the future holds for programmers and vibe coders. The layoffs are coming hard and fast. I happen to think it has a lot more to do with cost savings and convenience than AI. But it’s also true that software development has changed forever, like it or not.

Let me know what you’re worried about regarding AI.

• Not to jump to another job.

• Not to start a company.

• But to take a step back from everything.

Kevin shares his full journey—from growing up in the suburbs of Atlanta to building a career at Apple, and ultimately reaching the point where he could walk away financially and mentally.

You can follow along with Kevin below.

Unmapped

Left Apple after ~13 years to backpack 1,000 miles. What happens when you stop optimizing and let the trail shape what’s next?

By Kevin

We dive deep into what it’s really like working in tech: the high salaries, the lifestyle creep, the pressure, and the surprising reality that even people making great money often have no clear financial plan.

This conversation also explores the rise of FIRE (Financial Independence, Retire Early), how Kevin discovered it through Mr. Money Mustache, and why his perspective on it has changed over time.

What starts as a path to freedom can easily turn into a scarcity mindset—and that’s something most people don’t talk about.

We also get into:

• Why high income does not equal financial freedom

• The hidden trap of lifestyle inflation in tech

• The simple investing strategy that actually works (and why most people ignore it)

• Why many engineers are “close” to freedom—but never pull the trigger

• The psychology of money, status, and why people stay stuck

• How a failed project and burnout became a turning point

• And how Kevin went from overworked and unhealthy… to climbing mountains and preparing to backpack 1,000 miles

This is not your typical “get rich quick” or “retire at 30” conversation. It’s a grounded, honest look at money, work, and what it actually takes to build a life you don’t need to escape from.

Some time ago, I used my power of divination after decades of writing online to anger the self-righteous pundits of Reddit fame with a critical take on Notebook Engineering. I’ve got their number dialed in. I know human nature and what gets the blood boiling, and yes, I use it to my advantage.

Hey, I’m just human after all, what do you want from me? I’m a slave to my own passions.

An astute reader reached out to me with a very good question, and a difficult one at that, arguably the million-dollar question of many a Data Team.

The only downside to this very to-the-point question is that I can’t bumble and mumble my way out of a good answer. I mean, is that all I’m good for? Casting stones at unsuspecting victims?

I would hate for people to think I’ve turned over a new leaf and to offer practical, helpful advice.

This issue is sponsored by Thesys Agent Builder

It helps support his Newsletter by clicking the link below and supporting our generous sponsors who make this content possible.

Build a data insights copilot in 5 minutes. I just uploaded a spreadsheet and started asking questions.

- No SQL.
- No dashboard building.
- No text-heavy LLM responses pretending to be analysis.

Thesys Agent Builder lets you upload a CSV/XLSX (or connect your database), ask plain-language questions like:

“What are the top-performing regions this quarter?”
“Compare this month’s revenue to last month.”

• It’s a shift from dashboards → conversations.

• From static reporting → generative UI.

• From text answers → visuals you can act on.

Try Thesys Agent Builder — build your own data insights agent in 5 minutes.

Breaking free from Notebook Engineering.

So, you and I, we're gonna have a chat … I'm gonna have a real-life no-bull discussion about hard topics. I truly hope you don’t get your poor feelings hurt and go running to Ma and tattle on me. Do what you have to; I can’t take it.

Methinks that a Notebook Engineer is more of a way of life. Is it not?

It’s an identity, it’s who someone is. Or a team. Or a company.

What a “Notebook Engineer” is

A notebook engineer is someone who uses tools like Jupyter or Databricks notebooks 
as their primary way of building, running, and even deploying data workflows—often 
far beyond what notebooks were originally intended for.

Look. There is no longer any excuse. You don’t even have to write the flipping code yourself anymore. You can one-shot your way to Claude or Cursor glory while lying on the couch scrolling through Netflix. Don’t lie, I see you.

• The main problem is that the Notebook Engineering mindset follows you down the AI path and probably just amplifies the bad habits you have already adopted.

This is why a junior developer armed with Claude Code might ship more while still being a Junior Developer. Yes, with the capital letters and all. Everything has changed, and nothing has changed. Code was never how someone got promoted to Senior+ levels in tech.

Share

Here is my list of ways people can break free from Notebook Engineering … but doing the OPPOSITE of what I list below. These are the traits of someone trapped in Notebook Engineering.

Traits of Notebook Engineering Mindset

• They do most (or all) development in notebooks rather than in proper codebases, scripts, or structured pipelines.

  • It is no longer a helper or an exploratory tool; it’s a way of life.

• They treat notebooks not just as a place to explore data, but as the final implementation, even in production environments.

• Optimizes for speed and ease, not rigor

• Avoids (or lacks) traditional SWE practices

• Testing is never done, or rarely done

• There is no system design or thinking. They see the world as one giant Notebook attached to Serverless.

• Common gaps:

  • testing

  • modular code

  • version control hygiene

  • reproducibility

Save me your speech on how you unit-test your Python Notebooks. I believe probably 1 out of 100 people do that. The truth is, there are no more excuses, not with Cursor and Claude becoming our fingers.

Do you struggle with writing code that reflects software best practices?

There is almost nothing AI is better at than writing unit tests for you, making your code clean and modular, heck, it’s great at setting up CI/CD pipelines and the rest. All that busy work that kept you from doing the right thing in the first place, you don’t even have to do it yourself anymore!

Read more

Sometimes I lean back in my chair, before work starts, and dream of the time long ago, before AI ruined everything. Back in the olden days when simplicity was > than complexity. It seems those days are long gone.

People all of a sudden worship at the altar of Claudius Maximus Spit Codeius. Overnight, for the most part, all the classic dos and don’ts of software development best practices went out the window. The faster you can spit the code, and the more of it you spit … the better.

Back a few years ago …

• We measured success by the ruler 📏 of doing the job with the least amount of code possible.

I fear the Claude Apostles have never thought about this. Heck, why would you? The creation of code has been commoditized to the nth degree. AI writes, maintains, tests, and debugs itself. I giant and most classic of infinite loops, while we look on in wonder.

In an ode to the old and boring days, we return to the most simplistic of subjects. Service Level Agreements (also known as SLAs) to CTOs.

Share

Airflow and SLAs

What are two of the most common “things” we run into in the “data space.” Shouldn’t be that hard to guess.

• Apache Airflow

• SLAs (“Why is my data not available yet??!!”)

Airflow has long been ubiquitous as the de facto orchestration and Dependency tool of this decade. Even AWS and GCP offer Airflow as a managed service. It’s the tool to beat for all the upstarts. I recently wrote about Airflow and did the survey below. I’m surprised that it wasn’t more than 50% of folks using Airflow.

That being said, we all have a love-hate relationship with the tools (and people) we find ourselves with all day long. We are starry-eyed in the beginning, and jaded at the end. Such is most people’s relationship with Airflow. We love it for its simplicity and familiarity; we hate it for the obvious things we fight over.

That brings us to today’s topic.

Service Level Agreements (SLAs) for Apache Airflow DAGs, pipelines … whatever you want to call them.

An SLA stands for Service Level Agreement.

It’s a formal contract (or part of a contract) between a service provider 
and a customer that defines the expected level of service.

In simple terms:

An SLA answers:
👉 “What level of service should I expect, and what happens if I don’t get it?”

We work in a real and broken data world. Pipelines file, for who knows what reason, something that was running fine for the last 6 months decides to puke or crawl at turtle pace, with no end in sight, four hours later.

Slack starts to blow up, emails from customers start coming in… your manager asks you what you did. We needed that data 3 hours ago, and the pipeline has been running for 5.

So, what’s a vibe coder to do?

Subscribe now

Simplistic SLA checks for Apache Airflow Pipelines.

If you run a majority Apache Airflow orchestration platform, and it drives most of your pipelines, one could assume that many of those DAGs and outputs thereof are directly tied to end products (internal or external) that someone, somewhere, is expecting to show up at some specific time every day, hour, month, whatever.

This might be a task in itself, simply understanding the relationship of DAG(s) and Pipeline(s) to …

• End products

• What the SLAs are for those products

The first order of business is always documenting those data products, who owns them, who the consumer is, and getting the business (or consumer) to explicitly tell you what the SLA is.

Often, there can be a difference in the Engineering perceived SLA requirement, if it even exists, and what the business says is the actual need.

Heck, if you can end up with a list of data products and SLAs (who needs what by when), you’re probably ahead of 80% of Data Teams at that point already, and all you’ve done is document.

marketing data product #1 -> 7 AM, Monday-Friday
marketing data product #2 -> 10 AM, Wednesday
C-suite data product #1 -> Hourly, Monday-Friday
...

Even though this is the least technical task, in the age of AI, this kind of knowledge is extremely useful and sets A-Teams apart from B-Teams. It’s the difference between burnt-out teams and those that sleep soundly at night.

Why?

I don’t know how else to put it.

There are errors, and then there are errors. To a data team that has no idea which SLAs are tied to which data products, and therefore to which data pipelines (in this case, Airflow DAGs), every error or failure is cause for panic and mayhem at all hours of the day and night.

The truth is, much heartache and tears can be avoided with simple knowledge.

So, let’s say you’re a unicorn and you actually gather this data, you now have to bridge the gap between theory and reality (Apache Airflow DAG reality).

Share

Airflow SLA DAG

I’m very curious about how you would solve this problem, or how you have solved it in the past. I imagine every data team is different, with different notifications, DAGs, pipelines, and SLAs.

Before reading further, leave a comment: how would you deal with this problem?

Leave a comment

We can go complicated or easy. I typically think it’s best to waterfall your way into a solution. You don’t have to go scorched earth immediately. Sometimes the tortoise does win the race, and not the hare.

Maybe we can define a very simple DAG like this. Nothing easier. Just put a schedule on it.

From there, our custom Python logic (PythonOperator) could be unique to each of us, depending on our requirements. Maybe we just want notifications when certain DAGs exceed the runtimes we care about, based on our SLAs.

I mean, I’m sure Claude can spit out whatever logic you would like, complex or simple, like this.

Maybe you decide to keep all the SLAs in a database table or something that can be pulled in and looped through, who knows. Either way, we are taking steps in the right direction, are we not?

It really isn’t about the exact logic and Python code we use in our Airflow DAG to check on all those other pipelines and DAGs. That is probably the easy part.

• Pulling DAG metadata from Aiflow isn’t a hard problem to solve.

• Comparing that metadata to our SLA isn’t really that hard either.

Airflow provides plenty of pre-built methods to interact with DAG state. It isn’t really a problem of “What state is my pipeline in, how long has it been running, when did it start, when did it stop?”

Over the years, seeing a number of different Data Platforms and teams, it’s always the last 10% that makes a difference.

We all run pipelines, monitor stuff, etc. But who’s going the extra mile to even document every single data product, let alone the SLA assigned to each data product … and then to go the last few steps to monitor those SLAs.

I get it, I’m sure there’s plenty of SaaS companies willing to sign you into a contract to do some fancy monitoring. Or you could just do the work to understand your data and its consumers. There’s that.

Then you could just write a few lines of Python code and send an alert to some Slack channel when something is actually going wrong.

You can’t go wrong with …

• simple

• more knowledge on your datasets and SLAs.

In this episode of Data Engineering Central, I sit down with the founder of DataFlint, , to talk about the realities of working with Apache Spark, distributed data systems, and the future of data engineering.

We start with his early journey into tech—how he first discovered large-scale data systems and the lessons he learned from working with real-world Spark workloads.

• The conversation then turns toward the future of data engineering, particularly the growing role of AI in software development and data infrastructure. We discuss why generic AI coding assistants often struggle with complex distributed systems, whether AI will eventually be able to automatically optimize data pipelines, and how the role of the data engineer may evolve in the coming years.

We covered a lot of career advice for new and upcoming data professionals.

We also discuss the origin of DataFlint, a tool designed to help engineers better understand and optimize Spark workloads by analyzing execution plans, logs, and runtime context.

If you work with Spark, large-scale data pipelines, or modern data platforms, this conversation will give you a deeper look into how the data engineering landscape is evolving.

Sometimes I’m not sure whether I should just sell all my belongings, move to a cabin in the woods, and raise goats, or keep this Data Engineering thing going a while longer. The problem is, if you stay too long (like me), you're liable to start seeing history repeat itself a little bit.

Methinks we should basically put all data folk into one of two baskets, for easier counting and general relatabiliity. Sorta like a nursing home of sorts for the data community.

I should have seen this one coming, I didn’t, but I should have. With announcements and features like Stored Procedures on Spark/Databricks hitting the block, only an idiot like me would not have seen “transactions” coming down the pike.

A sure sign of my age and senile ways.

Well, all you youngsters sucking on the teat of Apache Iceberg and Delta Lake might say, “But we already have transactions in our Lake House architecture. We do INSERT, UPDATE, and DELETE while you’re still stretching your old bones in the morning.” Indeed, my friend, you do have something like a transaction.

But all those decrepit DBAs turned Data Engineers know better. I see you.

Share

Days of Yore and SQL Server have come home to roost.

Read more

Matt has spent decades building and architecting data platforms across technologies such as SQL Server, Oracle, DB2, Hadoop, Redshift, and BigQuery, and now focuses on modern tools such as DuckDB and single-node analytics.

We discuss how the data industry has evolved, what actually makes data platforms succeed, and where tools like DuckDB, Polars, Databricks, and Snowflake fit into the future of analytics.

We also dive into the impact of AI on coding and data engineering, and whether distributed compute clusters will remain dominant — or if more workloads will move toward high-performance single-node systems.

Topics Covered

• Matt’s early career and journey into data engineering

• The evolution of data warehousing and ETL frameworks

• Traditional enterprise data systems vs modern cloud platforms

• DuckDB and the rise of single-node analytics

• Polars vs DuckDB: where each tool shines

• Databricks vs Snowflake

• AI-assisted coding and its impact on engineers

• The current data engineering job market

• Lessons learned from decades of building data systems

• Writing a book on DuckDB

Matt Martin's Substack

Matt's substack on high performance data engineering topics

Share

If there’s one tool I don’t think gets enough love, some, but not enough, in our weird data world, it’s Polars. I’ve been writing about and using Polars off and on since about 2022, so it's been a bit. It’s the first tool I used to replace a Databricks Spark job in Production, so it will always hold a special place in my heart.

I’m still a believer in the Single Node Rebellion, as the data community at large refocuses on the costs surrounding the Data Platform amid uncertain economic times. I do hope Polars will play an increasingly large role in the modern data stack; there is no reason why it should not.

Polars (proper) should take a note from the shining north star of DuckDB and MotherDuck, who spread the love of data and community far and wide, always with a smile and a gentle nod. All you get from the Polars ringwraiths is howling and nashing of teeth.

Curiosity gets the best of me, how many of y’all are using Polars in production today?

I’m not sure how many of you Claude-ites are used to using Polars, so I wanted to revisit it, hoping to convince some of you to reach for it when explaining to Claude how you want your next data pipeline built.

I see you, you vibe coding little stinker.

Today’s sponsor is Buoyant Data. Buoyant Data can help your team optimize bronze and silver so you can focus on the gold.

“Excessive platform cost turns data from an asset into a liability. Buoyant Data can optimize your ingestion and transformation architecture so you can spend less money getting the data in, and more time getting the value out.”

Read more

We talk about how he first got into software, the early jobs and tools that shaped his thinking, and the massive technology shifts he’s witnessed across decades of engineering—from early stacks and tools to today’s AI-assisted workflows.

• We also dive into the difference between coding and real-world software engineering, what separates junior, senior, and principal engineers, and why many developers misunderstand what it takes to grow in this field.

• We discuss leadership vs individual contributor paths, the origin of his Coding Challenges platform, why algorithm puzzles dominate developer culture, and what actually makes engineers improve quickly.

Share

Finally, we tackle the big question everyone is asking right now: how AI is reshaping software engineering, and what skills will matter most over the next decade.

Coding Challenges

A weekly Coding Challenge to help software engineers level up.

By John Crickett

If you’re anything like me, you’re tired of the never-ending stream of useless AI content that seems to have reached a fever pitch. Echo chamber upon echo chamber, influencers convinced they have reached a new level of Agnetic Nirvana; they sip tea while a swarm of digital workers produces the most perfect code.

It’s the same old delusions of programmers focused on the wrong thing, convinced their custom Neovim setup has somehow catapulted them into god-level status, and destined them to be worshipped as the premier coder of our age.

The same goes to the Claude and Cursor neophyties, conviced their Skills have opened up a new horizion of consciousness.

I mostly ignore that crap. Just as I ignored the “Checkout my IDE from heaven” acolytes of the past.” True careers were never built upon code, before or during the Age of AI, but on the application of technical skills to the surrounding ancillary and adjacent sources of pain … called the business.

I have simply adopted AI into my dev workflows as a nessary adoption of 
technology that the market place demands. It's truely a game changing
set of tools that can make working throughout the entire software lifecycle
from concept to reality, a pleasure.

I’ve also had the blessing, or the curse, of building real-life agentic systems that actually have to be more than “Hello, world.” The ChatBot tutorials simply don’t cut it when the rubber meets the road, and you have to meet the ever-changing, demanding business needs.

Share

Read more

It’s been a while since I’ve dredged the corners of the inter-webs to find the interesting and worthwhile content. Now that ChatGPT is responsible for most of the glop plopped onto LinkedIn articles, finding real, authentic content is more valuable than ever.

It’s been about 5 years or more since I’ve written “Top Blogs to Follow” post, so let’s do a refresh.

Let me preface this by saying this is MY favorite top Data Engineering blogs that I FOLLOW. We might have different tastes and be interested in other things; we are distinct, you and I. It’s also important to me that the content is posted regularly, once every few months is rarely going to make my list.

I am going to try to put them into a few different categories to help summarize what we are dealing with and how I think about consuming content.

• Thought Leadership

• Corporate Engineering Blogs

• Simple Data Engineering Content

• Analytics

• General Programming

Top Data Engineering Blogs and Newsletters to Follow

So without further ado, and in no particular order, I’m just going to list the categories, the blogs in each category, and WHY I read them.

Thought Leadership

• joereis.substack.com

  • is the author of the most popular data engineering book and has many thoughts on the data world and community.

• seattledataguy.substack.com

  • You’re pretty much an idiot if you work in data and don’t follow

• https://www.jesse-anderson.com/category/blog/

  • A true OG of the data space

• https://rmoff.info

  • Robin is a cool and thoughtful dude 😎

• substack.thehustlingengineer.com

  • on how to be a good engineer

• alifeengineered.substack.com

  • How to rise in the ranks

• read.highgrowthengineer.com

  • has one of the best Substacks in the land.

• alifeengineered.substack.com

  • the best content of all time by

• newsletter.eng-leadership.com

  • Learn leadership by

• datagibberish.com

  • Hard-hitting content by

Corporate Engineering Blogs

• https://netflixtechblog.com/

• https://www.daft.ai/blog

• https://medium.com/airbnb-engineering/subpage/data

• https://eng.uber.com/

• https://databricks.com/blog/category/engineering

• https://www.linkedin.com/blog/engineering

Simple Data Engineering Content

• vutr.substack.com

  • deep dives into a plethora of Data Engineering content.

• dataengineeringweekly.com

  • has the GOAT Substack that gives a general overview of what’s happening in the data space and what others are writing about.

• blog.mehdio.com

  • Substack written by one of the premier data personalities of our decade .

• luminousmen.substack.com

  • Incredible depth in complex data engineering topics

• https://www.waitingforcode.com/

  • Very technical with depth.

• blog.dataexpert.io

  • that OG of Data Engineering , too the moon.

• robertsahlin.substack.com

  • good stuff by robertsahlin

• juhache.substack.com

  • Bro comes up with everything

• junaideffendi.com

  • Just plain old good data engineering content by

• pipeline2insights.substack.com

  • General Data Engineering by

Analytics

• learnanalyticsengineering.substack.com

  • The GOAT for real-life analytics content

• roundup.getdbt.com

  • Just stuff

• databricksters.com

  • Databricks stuff

• practicaldatamodeling.substack.com

  • Another good one by
    

General Programming

• newsletter.pragmaticengineer.com

  • What’s happening in the great world of programming in general.

• codingchallenges.substack.com

  • is actually the smartest person on the internet

• https://towardsdatascience.com/

  • just general stuff

• thesweekly.com

  • good programming stuff by

• blog.bytebytego.com

  • learn everything from

• packagemain.tech

  • general programming thoughts

I know this probably barely scratches the surface, but that’s about all I can reasonably keep up on in my spare time. I try to focus on finding non-AI-generated content that has a hard edge or lots of real-life content, not pie-in-the-sky stuff that has no real meaning in daily data engineering life.

Feel free to add suggestions in the comments if there is relevant stuff.

Leave a comment

We cover:

• How Yuki got into tech
• Early career lessons and pivots
• Tools and technologies he’s worked with over the years
• How data engineering has evolved
• The impact of AI on software development
• What engineers should focus on right now
• Advice for those building their careers in data

Yuki shares practical insights on navigating the industry, staying adaptable, and thinking long-term about your technical growth.

If you’re a data engineer, aspiring engineer, or just interested in where AI and modern software are going, this one’s for you.

Yuki writes on …

LinkedIn - https://www.linkedin.com/in/yukikakegawa/

https://yukikakegawa.me/#blog

The Data Toolbox

Your go-to guide for modern data tools, empowering data professionals to stay ahead and choose wisely.

By Yuki

Struggling Dad

A Japanese dad navigating fatherhood in the US, one day at a time

By Yuki

The Independent Insight

A data guy shares his learnings while building a successful independent consulting business.

By Yuki

🔔 Subscribe for more interviews with leaders in data engineering, AI, and modern data platforms.

Subscribe now

So, recently, I had the pleasure of poking and prodding that new kid on the block, Agent Bricks from Databricks. We live in the age of Agents, that much is clear, in case you’ve missed the news. You can be an AI skeptic or lover; I care not, but Agents are coming and are already solving and automating all sorts of use cases.

I’m no AI doomer; I think the future is bright for good Data Engineers. We build Data Platforms and massage the data that enables such tools to be built at scale.

Everyone is pumping out their Agentic tooling and frameworks as fast as those engine… I mean, Claude can spit out that code. Agentic AI is becoming commoditized; there’s no longer a technical barrier to building AI systems.

• The question is, can something useful be built?

Talk about ease of use, building Agents, and Agent Bricks fulfills that promise.

If you’ve been reading my content for any length of time, you will know that I’ve given a reasonable tip of the hat to all things AI, Agents, ChatBots, Vectors, LLMs, and other such tomfoolery. Call it self-preservation if you want, but I keep my options open.

• LLMs for PDFs

• Fine Tuning Ollama - Part 2

• Demystifying LLMs

• What is TOON?

• Ollama + SQLite + Langchain

• Building Agentic AI

• Vector Databases

• Lance for Embeddings

• …

Yeah, yeah, I hear ya, “Stop your jabbering and get to it, we want some Agents!” Jeez, fine.

A one-day conference for data engineers and architects.
- Speakers to include Joe Reis
- No vendors, no salesfolk
- Completely free
- March 26, in SF
- Limited to 100 attendees; register for free here

Agent Bricks from … Databricks.

Subscribe now

We have reached the point in our collective AI/LLM journey that the pace of innovation, if you want to call it that, is hard to keep up with. It’s easy to get suck in the echo chamber and get caught up in the marketing hype of people who’ve got some money or skin in the game.

The future of Data Engineering lies in being able to build, and speak to, the infastructure that underpins Agentic systems.

So, back to Agent Bricks, Databricks provides a whole host of “out-of-the-box” Agents that can be built from within the UI. Talk about approachable, eh??

Agent Bricks consist of the following categories to choose from …

Document Parsing
- Parse and visualize document structure with AI.

Information Extraction
- Extract key information and insights into structured JSON.

Knowledge Assistant
- Turn your docs into an expert AI Chatbot.

AI/BI Genie
- Turn your tables into an expert AI Chatbot.

Supervisor Agent
- Design a AI system with Genie, agents, tools.

Custom LLM
- Specialize an LLM to perform custom text tasks.

Code Your Own Agent
- Build with OOS libraries and Agent Frameworks. 

This makes perfect sense. You have the ability to build pretty much any Agentic system you want, custom or otherwise, within these categories, including multi-Agent systems.

One could ask, so what’s so special about this, big deal. Yes, and no. The compaines that will win the AI and Agentic race to the bottom, are the one’s that simplify and obvuscate away the tricky parts … aka … the infastructure.

This is one thing Databricks is good at: think about how they brought Spark to the masses and made it as easy as a few clicks. All they have done is apply that to Agents.

Building a Knowledge Assistant.

Maybe you haven’t had the chance to build any Agents in a production-like environment, no personal projects to work on, relegated to being an onlooker to those new-fangled AI Engineers doing all the fun stuff.

Well, never fear, we can still learn and build things.

• I will use my personal Databricks account (you can have one too, for a few bucks a month) and build a Knowledge Assistant Agent using a few of my blog posts. I’m sure you have some PDFs or something lying around, or could think of a similar use case.

What we really have is an obfuscated RAG system.

This is probably one of the most under-appreciated Agent systems yet, one that most businesses would benefit from building. A wealth of internal knowledge, once documented, became a 24/7 Agentic Expert.

Building a Knowledge Assistant with Agent Bricks is simple and effective.

• Build a Databricks Volume or Cloud Storage with our documents

• Built a Knowledge Assistant

• Sync our documents to the Knowledge Assistant

• Done

A couple of side notes, although this is constantly changing.

1. txt, pdf, md, ppt/pptx, and doc/docx.
2. Files larger than 50MB won't be indexes.
3. The first inital sync can take some time.
4. You can an adjust the agent's behavior based on natural language feedback.
5. You can import pre-labled datasets

Basically, you’ve got everything you need here to build a reliable and easy-to-use Knowledge Assistant (RAG) that can be used in and of itself or integrated into a multi-agent system!

Create the Knowledge Base

Ok, so let’s see how easy it is to create a managed Databricks Volume to store some documents, in my case, a few blog posts as TXT files.

Remember, as we do this simple exercise, we are thinking bigger picture and trying to learn from what we are doing. In a real production system (I’ve built ones that look similar to this), we might have all sorts of internal knowledge docs in various forms that wewould need to gather and store in a logical way.

• This is a problem in and of itself, managing and organizing large amounts of various documents, pdf, txt, doc, ppx, etc.

A Knowledge Assistant is only as good as the work put into it. You need to spend the time to gather, or write down, knowledge that has been spread around … gather it into a single place where it can be managed … and then connect it into Agentic systems. This is no small feat.

Ok, so we have our docs in a Managed Databricks Volume. Let’s create our Knowledge Assistant and connect our data.

Subscribe now

Create the Knowledge Assistant

Now the important and easy part, where the magic happens. If this part is simple and uncomplicated, that is a good thing. It shows how far we have come in a few short years in building RAG and Agentic systems that focus on delivering value rather than complex infrastructure.

I told you this would be the easy part. Click a few buttons, point it at our Volume, which serves as the source of our knowledge, and wait for the sync to complete.

Subscribe now

Once we have built the Agent, there is nothing more much to do, besides put it to use. You can add optional instructions to let the Knowledge Assistant know how you expect it to act.

Also, you can chat with the Knowledge Assistant in the Databricks Playground. You can see here that I asked about Databricks cost savings; it pulls one of the blogs I wrote and summarizes that information for me.

You know the other great part? We can interact with this Agent via Python and an Endpoint.

Of course, we could integrate this ourselves or use an Agent Bricks Supervisor Agent for some multi-agent workflow if we wanted. Easy as pie.

Look, I know everyone wants to look cool and make everything seem to be more complicated than it is. And sure, if we were having this conversation 2 years ago, it would have been much harder and more involved to build RAG systems.

Don’t get me wrong, you can still build Agentic systems the hard way, as I have done recently.

But AI infrastructure and systems are just like any other technology we deal with. Over time, as things solidify in the community and frameworks, things start to harden and become more “commoditized.”

Agent Bricks is the perfect example of that. The barrier to building helpful and useful Agentic systems is so low that it’s hard to understand why people are not building useful tools.

Knowledge Assistants (classic RAG) are among the easiest and most obvious entry points for using AI in the business. Every team and company has a lot of knowledge scattered across different places and brains. There were enough excuses a few years ago, but now it’s almost TOO easy to build these Agents.

What I also want you to think about is the infrastructure and engineering behind all of this.

• Building “Chat” interfaces

• Serving RAG and LLM as endpoints

• The compute required

• Building storage for knowledge bases

• Managing and cataloging documents

• Embedding documents

• Instructions and context

• Observability and monitoring

… of course the list goes on.

Agentic AI is within reach and not rocket science

I encourage you to explore these different Agentic offerings, Agent Bricks, and others. Try to understand what it takes to collect, gather, and massage data into usage formats that can be ingested into these AI systems.

Understand the pros and cons of each approach. What infrastructure is required for each layer of the AI workload? What tools, frameworks, and companies can take away some of the burden from each layer? What price do you pay for such ease? Do you loose flexbiliity?

We live in a brave new world full of exciting new technologies. It’s not the time to shy away or stick our collective heads in the mud, but to push forward into the future.

Subscribe now

If you are need of consulting help with implementing, upgrading, or solving your Databricks, or any data problems … feel free to reach out, I’m always happy to help!

This isn’t the first time I’ve written about that ever elusive Semantic Layer, that Death Star of analytics and data platforms, sucking in all calculations into a single spot. For anyone who’s spent some time around Data Platforms and Analytics, the pain of scattered, but important logic, is real pain.

Since time immemorial, data folks of all shapes and sizes have fought their way through a maze of GitHub repos, Confluence pages, and random scripts … in search of all the places that calculate x, y, or z.

One would think we’ve learned our lessons over the decades, but sadly, we are human, and to err is human.

I’m not here to tell you what to do or not to do. Semantic Layer, or Semantic Layer free Data Platform, that’s up to you. If indeed you’re tired of all the scattered business logic and calculations that never align or agree, and you want a solution … I give you Databricks Metric Views.

Subscribe now

This issue is sponsored by Thesys Agent Builder

It helps support his Newsletter by clicking the link below and supporting our generous sponsors who make this content possible.

Build a data insights copilot in 5 minutes. I just uploaded a spreadsheet and started asking questions.

- No SQL.
- No dashboard building.
- No text-heavy LLM responses pretending to be analysis.

Thesys Agent Builder lets you upload a CSV/XLSX (or connect your database), ask plain-language questions like:

“What are the top-performing regions this quarter?”
“Compare this month’s revenue to last month.”

And it responds with actual visual insights—bar charts, tables, summaries, not paragraphs. You can even generate presentation-ready reports and slides from the conversation in one click.

• It’s a shift from dashboards → conversations.

• From static reporting → generative UI.

• From text answers → visuals you can act on.

If you’re building with data but tired of navigating dashboards or writing ad hoc SQL, this is worth a look.

Try Thesys Agent Builder — build your own data insights agent in 5 minutes.

https://thesys.dev

All hail the Semantic Layer

I know when to say Uncle, you know what I mean. Sure, one can be a skeptic about this or that marketing schemes, but we have to take out the lumps, too. One can’t be too sure what will stick in the craw of the CTOs and Data Power Brokers, and be force-poured down the throats of unsuspecting Data Engineers.

No one fully agrees on what it is — but everyone seems to be selling one.

You compare how major vendors define the Semantic Layer:
- Databricks positions it as something that sits after data lakes/marts but before BI tools.
- DuckDB describes it more simply as something that sits after the database but before the business user.
- Snowflake actually ships a concrete implementation via “Semantic Views” — schema-level objects that define business metrics and entities.

The only consistent agreement across vendors:
- It sits between data and end users.
- It standardizes metrics and definitions.
- It often includes governance and permissions.

Sometimes includes transformations. Beyond that? Definitions diverge.

Anywho, if Databricks believes in the Semantic Layer, I might as well hitch my wagon to that horse. I try not to bite the hand that feeds me; I can read the handwriting on the wall.

The concept(s) at the core of the Semantic Layer are indeed on the high moral ground.

No matter how you dice it, code bases … human or AI developed have a tendency to fall prey to loose structure over time, sprawl overtakes, things get dirty and loosey goosey. Databricks Metric Views, as a Semantic Layer, aim to address that problem straightforwardly.

Share

Nuts and bolts of Metric Views

Let’s just get to it and poke a stick into Metric Views and see what crawls out. It might be fun, or boring, but you know, that’s life. I think it’s important for all of us, at the very least, to understand what features and options are available to us as builders of Data Systems and Platforms.

You don’t know what you don’t know, and learning and knowing is half the battle. It’s good to practice our systems design and ensure our knowledge is up to date with the real world. This makes you different from Cursor or Claude spewing out bits and bytes. So, let’s understand Metric Views from the viewpoint of the Semantic Layer.

Here are some basic facts about Metric Views in Databricks.

• Built-in materialization if desired

• SQL based

• Supports complex JOINS

• The source can be a view, a table, or a SQL query

• Lineage available via the UI (what’s using the Metric View)

• Manage Permissions on Metric Views

Metric Views in Databricks are first-class semantic-layer objects that live in the Unity Catalog. They let you define governed, reusable business metrics and dimensions once, and then query them consistently across tools such as SQL, BI dashboards, and AI/BI experiences.

• They represent business logic and key performance indicators (KPIs) in a way that’s semantic (business-meaningful) rather than purely technical.

• Metric Views are stored in Unity Catalog and behave like views, but with enhanced semantic structure and metadata.

Nothing other than what you would expect. I guess the key would be understanding the materialization options, knowing that by default, the query behind the Metric View would have to be recalculated every time someone hits it. But if you are a SQL person, you are well familiar with Views and their ups and downs.

• Let’s make our own Metric View with Divvy Bike trips. This dataset contains a bunch of bike rides people took in different cities, along with their start and end locations.

So, as per normal, any sort of Semantic Layer is really about Analytics, with a capital A. That is the whole point: capturing business logic in a single spot for reuse.

Subscribe now

Of course, with Metric Views and Databricks, we get a whole host of features that come with Unity Catalog … permissions, who can do what, governance, lineage of usage and change tracking, and access to different Metric Views. This, of course, is a bigger deal, the larger the organization.

Let’s make a Metric View based on our raw bike trip data, just to see how easy it is.

My first Metric View might not be what you're expecting; that is on purpose.

CREATE OR REPLACE VIEW confessions.default.trips_metrics
  (ride_date COMMENT 'Date of trip (from started_at)',
   ride_month COMMENT 'Month bucket (from started_at)',
   member_type COMMENT 'member vs casual',
   rideable_type COMMENT 'Type of bike/ride',
   start_station COMMENT 'Start station name',
   end_station COMMENT 'End station name',
   route COMMENT 'Start → End route label',

   total_rides COMMENT 'Total rides',
   member_rides COMMENT 'Rides where member_casual = member',
   casual_rides COMMENT 'Rides where member_casual = casual',
   avg_trip_minutes COMMENT 'Average trip duration (minutes)',
   total_unique_start_stations COMMENT 'Proxy: unique start stations',
   total_unique_routes COMMENT 'Proxy: unique routes (start→end)')
  WITH METRICS
  LANGUAGE YAML
  COMMENT 'Metric View for trips: rides by day/month, membership split, popular stations/routes, durations.'
AS $$
version: 0.1
source: confessions.default.trips
filter: TO_TIMESTAMP(started_at) IS NOT NULL

dimensions:
  - name: ride_date
    expr: TO_DATE(TO_TIMESTAMP(started_at))
  - name: ride_month
    expr: DATE_TRUNC('MONTH', TO_TIMESTAMP(started_at))
  - name: member_type
    expr: member_casual
  - name: rideable_type
    expr: rideable_type
  - name: start_station
    expr: COALESCE(start_station_name, 'UNKNOWN')
  - name: end_station
    expr: COALESCE(end_station_name, 'UNKNOWN')
  - name: route
    expr: CONCAT(COALESCE(start_station_name, 'UNKNOWN'), ' → ', COALESCE(end_station_name, 'UNKNOWN'))

measures:
  - name: total_rides
    expr: COUNT(1)
  - name: member_rides
    expr: COUNT(1) FILTER (WHERE member_casual = 'member')
  - name: casual_rides
    expr: COUNT(1) FILTER (WHERE member_casual = 'casual')
  - name: avg_trip_minutes
    expr: |
      AVG(
        (UNIX_TIMESTAMP(TO_TIMESTAMP(ended_at)) - UNIX_TIMESTAMP(TO_TIMESTAMP(started_at))) / 60.0
      )
  - name: total_unique_start_stations
    expr: COUNT(DISTINCT start_station_id)
  - name: total_unique_routes
    expr: COUNT(DISTINCT CONCAT(COALESCE(start_station_id, 'UNK'), '->', COALESCE(end_station_id, 'UNK')))
$$;

Bet you weren’t expecting that. Metric Views as YAML, say it ain’t so, oh it’s so.

How is this helpful as a Semantic Layer? Well, you would have to agree with me that it’s extremely verbose and very forward about what a “Measure” is.

... metric definitions, which include dimensions and measures, based on 
a data source, or multiple sources if join logic is used ... A measure is a value 
that summarizes business activity, typically using an aggregate function 
such as SUM() or AVG(). 

• We can just be more explicit about defining all important business logic and calculations.

Instead of embedding this sort of logic WITHIN DATA PIPELINE(S) … which is usually what happens, we can just use Metric Views to create a single source of truth for this sort of logic, and require many data pipelines, teams, who or whatever … to get the answer from this single place.

Databricks has taken it even farther, looking forward to Agentic systems, like Genie, and enabling more business logic to be captured to help both humans and machines.

“Semantic metadata includes display names, format specifications, and synonyms that provide additional context.” - docs

For example, display names and synonyms.

The funny thing about business logic, KPIs, aggregates, analytics … whatever you want to call these data points that, many times, literally drive the business … is that the intersection of engineering and business is half art, half science.

Databricks Metric Views are a unique take, and the perfect Semantic Layer to capture not only the very technical details of a METRIC, how something is calculated, but also how the business refers to that idea, with features like synonyms. Amazing.

For most, it’s a mindset shift.

Personally, what I think doesn’t matter that much; it’s more about what the data community as a whole believes and how they act day to day when building and maintaining data platforms. Semantic Layers offer real benefits for organizations of all sizes, especially for medium- to large-sized teams.

Analytics, data feed back to the business, has for decades struggled to provide accurate results that matter, at least that is the common complaint, and it is warranted.

Many, not all, but many of the core problems stem from data teams at large being unable to manage their data, code, and logic coherently and consistently. It always devolves into the wild west, where numbers don’t match, and the business loses all trust in the data and the teams.

Databricks Metric Views, seen and used as a Semantic Layer to devour all business logic and calculations, will go a long way in solving at least some of these problems. Not all, but some.

The question is: if you don't use a Semantic Layer of sorts, how do you plan to manage your key metrics over the long term? How will your data team be any different than the legions that came before you, and dropped the data ball?

Share

Try Thesys Agent Builder — build your own data insights agent in 5 minutes.

We cover:

• How Bart got into tech and distributed systems

• His journey through different engineering roles

• Spark internals and why they still matter

• The realities of lakehouse architecture

• Streaming vs batch systems

• AI’s impact on data engineering

• What engineers should focus on in 2026

In a world obsessed with abstractions and AI tooling, we explore whether understanding the internals is still worth it — or if the game has fundamentally changed.

If you’re a data engineer, architect, or platform leader trying to navigate the next phase of the lakehouse era, this one’s for you.

—

🎙️ Data Engineering Central Podcast
Hosted by Daniel Beach

If you’re a CTO or data leader looking for help building or optimizing your data platform, reach out — consulting inquiries welcome.

Well, you know I simply can’t resist throwing my hat in the ring. If everyone else is going to stand on a soapbox and decry the end of all data pipelines as we know it, you can be sure I’m going to climb to the top of the old oak tree and scream my truth to the glassey eye’d masses who’ve been sitting in their echo chambers for far too long.

You, and everyone else are lying in bed at night, staring at the ceiling, wondering “Am I going to be fired tomorrow, replaced by Sky Net, now I will have to pick up empty pop cans on the highway to make ends meat.”

I don’t suppose this is particular to Data Engineering, fear of the unknown has plagued the human soul since the first caveman hit two rocks together and started a fire. When the iron horse made its way across the Great Plains into the West, those old cowboys probably were wondering what was coming next.

It’s hard to deal in hard facts when we live in the social media-driven age, with a glut of information … factual and otherwise. Is the recipe for the day. Clearly, AI is, at a minimum, a fundamental shift in culture and technology, shaking the core beliefs and truths we once held as Gospel.

Share

Attempting to level set ground truth in the age of AI.

I’m going to try to give a fair shake to every side of the argument, as to whether AI will destroy Data Engineering and Software as we know it. First, I think it’s helpful to step back from the situation and say outloud that …

• AI changes everything

• Fear mongering gets clicks

• Lots of people and money have skin in the game

• Changing fundamental ways of doing business takes time

• The worst case senarios in your mind rarely end up that way

The discussion about the intersection of AI and any job, including software, is fraught with emotions, both real and not-so-real. Everyone has an opinion, probably strongly held, and that’s fine. I encourage you to take the middle ground, that’s probably where the truth lies … not at either end of the spectrum.

We have the AI doomers that exist on either end of the spectrum. Typically, they are either creators wanting clicks, or someone who has some sort of horse in the race … directly related to you having certain feelins and interaction with AI.

• Hate AI and all it does

• AI can do anything and everything

If we strip away what we know today and what we've seen over the last few years of the AI arms race, I think we have enough experience and lessons learned to make reasonable assumptions about what the future could look like.

That being said, the heartburn is real.

“Recent Gartner surveys reveal widespread anxiety regarding AI, with 79% of 
professionals fearing workforce disruption. Only 26% of job 
candidates trust AI to evaluate them fairly, and 32% fear it may 
reject their applications. Additionally, 53% of consumers distrust AI-powered 
search results and summaries.” 
     - summary of Gartner

Subscribe now

Let’s be real, two things can be true at the same time. We’ve had, over the last few years, a softening economy that has affected tech. Interest rates, global uncertainty, conflict, and political unrest. Economic softness and jitters have been the meal served to us for some time now.

The big tech bubble from COVID has popped, and layoffs have been coming hard and fast for the last two years. Mind you, this started happening before, but it did overlap with the AI is going to eat everything crisis. Talk about bad timing.

It’s perfect if you are a fear peddler selling the next coding agent or skill to worried devs.

It’s also true that you’ve been fed the line for at least 2.5 years now, at full volume, that AI is going to take your software job. Yet. Here most of you still are, still writing code, ablbiet with Cursor and Claude … but here you are.

Why can’t two things be true at the same time?

• AI fundamentally changes software

• Not everyone loses their software job to AI

The less popular news, because it won’t get that many clicks, is the reality behind many pushes into AI. Let me share a little secret with you.

AI is technology. For multiple decades, companies have proved they are slow to adopt and easily fumble with new technology. Heck, even recent MIT studies show it isn’t all unicorns and sparkles.

What I’m trying to say there is a big gap, and risk, between the talking heads saying “Fire everyone, and use AI agents,” and that happening in the real world.

I’m not saying that some people won’t officially lose their jobs to AI. I’m sure it will happen here and there, but the chances of you losing your job to AI in the next five years are probably pretty close to zero.

Leave a comment

The dark side of AI eating jobs.

The truth is, there are always people at risk of losing their jobs to automation and tech. That is nothing new and has been happening for centuries. The world changes, jobs and businesses change to meet those new demands and take advantage of new ways of doing business.

Let’s think about it from a Software and Engineering point of view. What are the levels and types of dev workers, and who is at risk of being replaced by AI?

What is AI good at today, what is it getting better at, and how does RISK play into this equation? One cannot leave risk out of the equation, because doing so would be unrealistic.

CTO’s, Engineering Managers, Product Managers, high-level Architects, Staff/Principal Engineers. These people are going nowhere. In fact, one could argue that they will become high-demand jobs. When you are dealing with an AI Arms Race, you will probably try to hire more of these people.

• Senior+ Engineers are most likely to benefit from AI because they can be 10x’d with Cursor and Claude. You still need smart people to curate and handle AI power gently, with that human experience.

Could you make an argument that Mid-level and Junior Grugs will have a harder time not being replaced? Yes and no. I don’t think it’s that clean of a line. Why? Risk.

Subscribe now

How many companies are brave enough, in a digital world where their product depends on technology that doesn’t break, to simply throw expensive, hard-to-train software engineers out the window in favor of an AI Agent that has no accountability? Now that is real risk.

What would they be waiting for? All the talking heads say that moment has already arrived. All the marketing departments for the AI SaaS vendors have been saying that for two years. AI Agents can already do the work of these people, so fire them already, and pay us!

Do you know what has happened?

It seems that companies at large have shown great interest in AI Agents and their adoption. But there has not been some catostrphic apocolypic layoff of all software engineers.

Why? Because AI Adoption != Layoffs.

Heck, if you go on any job board, what do you see? Tons of jobs, and tons of new jobs. AI engineer jobs. Who would have thought that a new technology would lead to the need for a new set of software engineers? If we didn’t have our collective heads so far up the rear ends of the AI Fear Mongers, this would have been fairly obvious.

Look, I could be totally wrong; that is possible. I encourage you to leave a comment below and tell me what I’m missing, what I don't understand, or any real data I should look at.

Leave a comment

• I find myself after decades of munging data and code, a little jaded. Do I think there could be a reduction in the number of mid-level and junior dev hires? I think it’s possible.

If someone refuses to adopt AI, learn new things, they might get fired, but it will be because of their inability to learn and grow, not because an Agent took their job. I also think that, net-net, the new AI boom will lead to MORE software jobs, not less, overall.

Take a look at this data from a 2024 study by the Pragmatic Engineer and .

I mean, are people telling me that the line is going to totally reverse and go into freefall? I don’t think very many people really believe that.

Again, two things can be true at the same time.

• AI can change software in fundamental ways

• Human Software workers continue to be in demand

Sure, your job won’t look the same, but that’s ok. That would be boring. Change can be fun and exciting, but it can also be scary. That doesn’t always mean you'll get fired or laid off. You might, but it might just be normal economic cycles.

To anyone who’s been laid off or fired, because of AI or anything else, I feel for you, that sucks, and I hope you find the next thing soon, and it’s better than where you came from. Always look forward to the future; the past has lessons, but it’s best not to dwell too much.

Share

What then?

Heck, I’m not going to lie. I’m mostly an optimist and a realist when it comes to the intersection of AI and Software. I see my future in the dev world as secure, not %100 safe from normal economic cycles, but as safe as any other job.

The job boards appear to be full of data positions. I still get weekly LinkedIn InMails from recruiters. I’ve been told for over two years now that it’s all over, yet it’s not. In fact, I’ve been building Agents for work today.

What we are going to have Agents building Agents and talking to the business, understanding use cases, costs, architecture, and the nuances of what the CTO wants … and, with a single prompt, that all becomes production … serving customers and bringing in money without human intervention? Uhhhh … LOL!

• I say keep your head up! Be a glass-half-full kind of person. Embrace AI into your workflows, and learn the technologies that underpin it!

If you want some more food for thought, read these articles.

Happy trails, my friend, onward and forward! Let me know your thoughts in the comments below.

Leave a comment

I can see all you old-school SQL Server buggers just squirming in your seats, hot under the collar. How dare they speak of such sacred things, holy, without reference and a nod to the past? Methinks someone, somewhere, once said that there is nothing new under the sun. This indeed is true.

I’m sure there was a whole cadre of folk with a little grey in their hair, 
who, upon seeing the announcement of Temp Table support in Databricks, 
had to do a little double-take. I was right there with you. Ah, the memories.

It’s probably for the best.

Before all you cranky old DBAs turn up your noses and head for the hills, slow on down there. We live in a world of abstractions meant to ease the burden on pipeline writers, both human and otherwise.

I think you agree that anything that clarifies and simplifies the complexity of a data pipeline is worth a nod, at least.

A salute to the past, and the SQL-hungry engineers.

So, let’s talk about Temporary Tables in Databricks Spark SQL, the good, the bad (if there is any), and whatever we find in between. We can talk about what it is, when to use it, and WHY it’s even a thing.

“Temporary tables are session-scoped, physical Delta tables. They store data in an internal Unity Catalog location tied to the workspace. They use the same caching and performance features as standard Delta tables.” - docs

Here is my way of explaining what a Databricks SQL Temp Table is, who knows if I’m right, hopefully at least half right.

The concepts being …

- It is a physical Delta Table stored in Unity Catalog
- Only lives for as long as that Spark SQL Session
- Auto-cleaned up
- Can be treated like any other Delta Table in process/session.

It’s not exactly earth-shattering from a technical and usage perspective, don’t get me wrong, it’s THEE perfect feature for many pipelines … just easy to use.

Let’s see how it works, with a few lines of code. We can test this simply by attaching a Databricks Notebook to a SQL Serverless Warehouse … create some TEMPORARY TABLEs for currency, with fake customers and orders, and then in the last cell, just run a SQL query using them all.

Subscribe now

Yeah, well … I told you it wasn’t going to be that exciting, per se. Again, it’s not about how simple or complex the usage of a feature is; it’s what it OFFERS and CHANGES about how engineers interact with and write pipelines.

More or less, a TEMPORARY TABLE can be created as follows …

Simplicity is the key.

• “A clean-up service removes temporary tables automatically. This happens when the session ends.”

WHY should we care about TEMP tables in Databricks SQL?

It’s hard to pontificate on the reasons to use temp tables in Databricks SQL for those who’ve come from the SQL Server world of data pipelines for the days of yore. What we are talking about is … HOW people write data pipelines and encapsulate problems.

From Databricks' perspective, the answer is clear and valid.

“Many workloads from legacy data warehouses rely on temporary tables for staging and intermediate logic. Teams moving from these systems now keep familiar patterns without redesigning their pipeline … ”

It’s about enabling more people to use and migrate to Databricks/Spark workloads and data pipelines with less conceptual overhead. Databricks spells out the gains as …

• performance

• migration simplicity

• governance and isolation

• auto cleanup

Most data people are heavily reliant on SQL, and most of their pipelines and logic easily fit conceptually into a SQL mindset. Could you, in theory, not use temp tables, of course! Many (most) Databricks pipelines today DO NOT use temp tables, because this is a new feature.

Today, you have one of a few things happening …

• Folks create long-lived, permanent Delta Lake tables to store the “intermediate” data sources and logic.

• Possible “cache() or persist()” of Dataframes.

• Data pipelines are built with no need for conceptual intermediate datasets

  • These folks are probably not using SparkSQL much

The problem with using long-lived “normal” Delta Tables is that they quickly become messy, storage usage grows, and, typically, the tables used in this way are not well planned (e.g., with partitions or clustering).

Subscribe now

It’s very probable that the second two types of pipelines using cache() or persist(), or simply non-SparkSQL ETLs, are probably more classic Software Engineers working in Dataframe APIs and ML/AI spaces, where SQL is a smaller part of their workloads.

At the end of the day, temp tables are a whole cadre of data folk who write pipelines; they’ve done it for decades and years, and will continue to do so.

You can INSERT, MERGE, UPDATE, and DELETE … normal CRUD operations against these tables that only last a session can make complex data operations easier for SQL-based pipelines.

We should be transparent about any potential downsides of using temporary tables in Databricks SQL. The truth of the matter is that when humans are involved, you have to be ready for everything under the sun to happen, including downsides.

• “Databricks automatically reclaims storage in the background, usually within a few days.”

One can see a world where folk are creating TB+ temporary tables, or larger, because of a lack of forethought. At the same time, what about clustering and partitioning on those temp tables? All of this, combined with the table(s) not being cleaned up “for a few days,” could lead to an explosion in S3 costs.

With great power comes great responsibility.

As with all new features, there are many reasons to see the upside; temp tables are no exception. They will lower the conceptual barrier for many SQL-based teams migrating to and relying heavily on Databricks for data processing.

At the same time, when you give end users this sort of power, especially those with say … maybe not a solid data engineering understanding … like Analysts, PowerUsers, Data Scientist … it’s very possible you might have things happen you are not expecting.

Time will tell.

If you are need of consulting help with implementing, upgrading, or solving your Databricks, or any data problems … feel free to reach out, I’m always happy to help!

This isn’t a buzzword-heavy DevOps chat. It’s a grounded, honest discussion between two engineers about how people really get into tech, how careers evolve over time, and why modern infrastructure is as much about systems thinking and human judgment as it is about tools.

We talk through Maxine’s journey from early technical curiosity to hands-on DevOps work, dealing with “ClickOps” to automation-first infrastructure, and how writing and teaching reshaped the way she thinks about engineering.

What we cover in this episode:

• 🛠️ From ClickOps to DevOps — what that transition actually looks like in the real world

• 🧠 Why DevOps is fundamentally about systems and people, not just pipelines and YAML

• 📚 How Maxine went from self-teaching to authoring practical guides like LLMs for Humans and The DevOps Career Switch Blueprint

• 🤯 Common mistakes engineers make when learning DevOps, cloud, and distributed systems

• 🔍 Testing failures, production realities, and where modern infrastructure still breaks down

• 🤖 What AI and LLMs actually change for engineers, and what’s mostly hype

• 🧭 Career advice for engineers without a traditional background

• 🔮 Where DevOps and platform engineering are heading over the next 3–5 years

Throughout the conversation, Maxine brings a refreshing, human-centered perspective to topics that are often over-abstracted or oversold. We dig into the tradeoffs behind tooling choices, the reality of production systems, and the importance of learning how to think, not just what to deploy.

If you’re navigating a DevOps or infrastructure career, wrestling with modern stacks, or trying to make sense of AI’s role in engineering, this episode offers clarity, context, and hard-won insight.

Learn more about Maxine’s work:

• Writing & guides:

Ilovedevops

DevOps Engineer, Technical Writer, Author, FullStack Mom

By Maxine Meurer

• LinkedIn: https://www.linkedin.com/in/maxinemeurer/

• Gumroad resources: https://mameurer.gumroad.com