As I was winding my way through the various twists and turns of LinkedIn lately, strange it has become the least of the social media evils, by happenstance there popped a post about Comet, that Datafusion baby that’s been trying to take a corner out of Spark for a few years now.

I did poke a stick at Comet back in 2024, but it was a little painful. You can read about that experience blew if you’re interested.

One would have to assume with the increase in Comet chatter, that the project has gotten its sea legs under it and hopefully is a slightly better experience this time. I’m crossing my fingers at this point that I don’t have to build my own JAR this time.

I hope one can simply download the correct pre-built JAR for the Spark version we are interested in. That would be a step in the right direction.

Truth is I have do no “pre-work” when I sit down to write articles like this and explore or revisit tools. Endevoring to simply approach the problem space like the average engineer who interact with some new “thing” for the first time. I want to encounter that same experience and relay that to you. Save you some trouble.

I do not have particularly high hopes from what I might encounter on this foray into Mordor, based on past experience. Let me explain what I mean by this. There are two types of Engineers and tooling they build in the world …

1. Smart engineers building cool tools but lack critical 
developer-centric awareness.


2. Smart engineers building cools tools and understand 
developer-centric are the key to success.

This has been a problem as long as I’ve been (attempting) to write code for the last 20 years. To this day, it simply doesn’t change. Some tool and their ardent acolytes say “Look, we are faster, come unto us.” Yet, they never seem to crack the nut and become mainstream, and can’t figure out why.

All the while continuing to waste their resources on negativity, pulling down other people and tools in an effort to lift themselves into a place by pulling other things down. This never has worked, and never will, longterm, in any sphere in life.

I’ve been privy to see behind the curtains, meeting people at Databricks and MotherDuck (DuckDB), who are creating and running the creme-de-la-creme of data tooling. What makes them succesful?

• The have the fastest tool … NO

• The focus their energies on negativity and others .. NO

They simply have an almost irresistible set of qualities that is impossible to ignore or hid … or be pulled down in the mud by others.

• Positivity in all things

• Love what they do and other people

• Put Developer experience BEFORE everything else

• Serve and give back to the data community

So simple, but yet so hard for some folk to grasp. Why? I don’t know. It’s probably something inside them, unhappy people will always be bumbling along looking for people to suck down into their misery.

Let’s start what I’ve been avoiding

Ok, at some point we have to get to what we are after today, namely figuring out if Apache Datafusion Comet has improved itself over the few years since we last visited it. One thing is for sure. In the Year of Lord 2026 if someone is using Spark, they better be using Databricks.

I know there are still a few odd old-school curmudgeons banging around in the dark corners, BUT if you want to be taken seriously in the Spark world today, you must go where the users are. Databricks. Sorry. Not Sorry.

• Can we get Comet working on Databricks?

First, we need a JAR.

The GitHub page for Comet gives a semi-not-so-much clear instructions.

The installation guide gives us more info, including what Spark Versions we are allowed to use with Comet.

So, we will have to pick a DBR version that matches something in here. Strangely, they say “Published jar files are only available for released versions, ” which insinuates they have published, pre-built JARs for supported Spark versions, although there is no link or mention of where to find those JARs … one could assume Maven maybe?

• You would think linking and being obvious about where pre-built JARs are located for potential dev “customers” might be a fairly obvious thing to do. Or not.

Ok, let’s go find some JARs (hopefully), before falling back to building our own. Indeed, they have a Maven repo with the different Spark version Jars.

We should be able to line this up with a Databricks DBR version that fits our needs eh.

Let’s pick Spark 4.0.

So now we’ve got our comet-common-spark4.0_2.13-0.16.0.jar that we can align with DBR 17.3 LTS. Next, we just need to gather the required Spark configs needed to go along with this JAR, we can unwind what we need from the Comet examples given.

Based on what I can see, this is what we will need to add to our Databricks Cluster config.

--jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \
    --conf spark.plugins=org.apache.spark.CometPlugin \
    --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
    --conf spark.comet.explainFallback.enabled=true \

As well as possibly

--driver-class-path spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar

Also, if you want your brain to go numb, you can read through the Comet Configs located here, it’s the next page that comes up in the Installation Instructions.

I’m assuming some of these are important, and greatly assist with how Comet performs, but God knows how one is simply supposed to scroll through all 1000 of them and know what to do. Your guess is as good as mine.

• Might be nice if they put some sort of overview of the the most important ones, things that should not be left to default, or the most commonly used ones and put some sort of conceptual overview together for how to tune them.

To top it all off, there is a very large and complex “Compatibility” section detailing what Comet can, and cannot do, where it will fall back to Spark etc. This one will give you a headache as well.

If you meet someone behind the old oak tree at midnight, throw salt over your left shoulder, and say these words … “Ooga Booga” 3 times over. At that point you will know if the production Spark pipeline you have will benefit from Comet and how to tune the configs for optimal performance.

Tear* … let’s do this thing.

Ok, let’s switch over to Databricks and get our cluster setup. First, let’s get this JAR available somewhere Databricks clusters can access it. Probably a Volume would be the easiest way to do this.

Here I have made a volume Volumes/confessions/default/jars and put our JAR up in there.

Next, let’s make a little init.sh script that can run on any cluster.

#!/bin/bash
COMET_JAR="/Volumes/confessions/default/jars/comet-common-spark4.0_2.13-0.16.0.jar"
LOCAL_JAR="/databricks/jars/comet-common-spark4.0_2.13-0.16.0.jar"

cp "$COMET_JAR" "$LOCAL_JAR"
chmod 644 "$LOCAL_JAR"

Next, we can put that bash file into the same Databricks volume.

Ok, so let’s just use one of those dreaded All-Purpose clusters in Databricks, configure it with the init script, as well as the rest of our configs.

As well, we can add in that init script we made.

And finally, our Spark configs for Comet.

I have zero idea if this is going to work, just a shot in the dark. We will find out soon enough, maybe. Ok, next we need some data in s3 to munge around with. Let’s use the open-source Backblaze harddrive dataset.

We can get all the 2025 data and use that. This is 365 files for about 43GB of CSV data in total. Not big data, but then again, most Spark pipelines running today picking up CSV files at not munching on Big Data.

This will do for now. Let’s write some average Spark code that would represent the average Spark pipeline in production.

Read more

Well, it’s that time of year again. My feed has been full of a little bit of this, little bit of that from the Databricks Data and AI Summit 2026. It’s kinda hard to miss, although this year, I must say, it seemed a little on the quiet side as compared to others.

Who knows, 2026 has been a busy year in the world in all ya know, wars, rumors of wars, layoffs, and generally it’s either the beginning of a new world or the end of our current one.

Like any good hunter of technology, I keep my veteran ear to the ground, ruffling through the fluff and litter, trying to find out what is actually worth your time, and what is just another layer on the AI cake we are all too tired to choke down.

So I’m just going to give you my take on the important announcements and products released or announced at this years Databricks Data and AI Summit 2026. I mean I wasn’t there, so take me with a grain of salt.

Here’s my list, and I will then follow it up with my take on each one.

• Zach Wilson went after blowing a raspberry at them last year.

• Lakehouse//RT aka “Reyden”

• LTAP - OLAP + OLTP on a single copy of data in the lake

Yeah, I’m giving a big ho-hum to the plethora of AI shiny rocks released, like Genie Ontology, Genie ZeroOps, Omnigent, Unity AI Gateway, etc. The world of Agents and AI is still in flux, no clear winners, everyone is throwing stuff at the wall to see what sticks.

For my part, I’m a big believer in Databricks, they have a propensity to fundamentally change technology and how we use it. Truly ground breaking technical solutions and products they bring to market. If you can’t admit that, then check yourself. That being said, I’m just reviewing what I consider products or announcements that meet that critera. Ground breaking and game changing.

If you think I’m full of it about something, or I missed something big, drop a comment.

Zach showed up.

Hey, I like a little spice in my life, the hot stuff keeps everyone on their toes. I regularly get on the wrong side of powerful people, much to my joy. Nothing like free head space to bring me more followers.

Anywho, last year Zach was throwing rocks at Databricks, as you can see below. Good Lord you gotta love seeing behind the curtain sometimes. We need our own data Netflix series on this sorta thing ya know?

But, Zach attended this years Summit, which I was glad to see. I think Zach is one of the smartest and hard working engineers I’ve ever scene, and I love Databricks because they make the best products. I’m glad they are getting along now.

Anywho, enough on that, next.

Subscribe now

Lakehouse//RT: Databricks Brings Real-Time Analytics to the Lakehouse

This one caught my poor little ears as soon as the words dripped out onto LinkedIn. Again, all this stuff is new so who knows what the future holds or if anything will come of it, but it solves some major pain points we’ve been dealing with in a new and novel way.

• I’m excited for this one.

One of the biggest challenges in modern data architecture has been serving real-time applications from a Data Lake or Lake House. When we adopted technologies like Delta Lake and Apache Iceberg, aka file storage with ACID, we gained some stuff and lost others. Well lost isn’t a good word. Delta Lake + Spark ain’t no Postgres + Python ya know??

Traditional LakeHouses excel at ETL, analytics, machine learning, and business intelligence, but when teams need dashboards that refresh in milliseconds or applications serving thousands of users simultaneously, they often introduce an entirely separate serving database such as ClickHouse, Pinot, Druid, or Redis.

That second system brings another copy of the data, another synchronization pipeline, another set of security policies, and another operational burden.

It brings complexity and overhead. These technologies just had a difficult time jiving.

Lakehouse//RT is Databricks’ attempt to eliminate that architecture altogether. Instead of exporting data into a specialized serving database, Lakehouse//RT delivers millisecond query performance directly against Delta Lake while keeping the data inside the governed Lakehouse.

What is Lakehouse//RT?

Lakehouse//RT is a new real-time compute designed specifically for workloads that require both very low latency and extremely high concurrency. It is powered by a new execution engine called Reyden, which Databricks says was built from the ground up for operational analytics, application serving, observability, dashboards, and AI agents.

I’m still unsure if we are dealing with two things, or one … read this closely …

• Powered by Reyden, a “new engine for realtime workloads”

• Lakehouse/RT, “real time data warehouse”

Are these two separate things you can use, or you use them both at the same time? I don’t know. Time will tell the details.

Unlike traditional analytical warehouses that optimize long-running reports, Reyden focuses on thousands of simultaneous interactive queries while maintaining consistent response times. According to Databricks, preview customers have seen:

• Up to 16× faster performance than existing real-time serving layers

• Query latencies as low as 10 milliseconds

• Sub-100 ms performance on much larger datasets

• Around 12,000 queries per second while maintaining low latency

Eliminating the Serving Layer

The biggest architectural shift isn’t simply that queries are faster, we’ve been hearing people say “My thing is faster,” for a decade or more. The difference is that Databricks wants to remove an entire layer of infrastructure.

Today’s architecture often looks like this:

Delta Lake
      │
   ETL / CDC
      │
ClickHouse / Pinot / Druid / Redis
      │
 Applications & Dashboards

With Lakehouse//RT, Databricks wants applications to query the Lakehouse directly:

Delta Lake
      │
Lakehouse//RT
      │
 Applications
 Dashboards
 AI Agents

That means:

• no duplicate storage

• no synchronization pipelines

• no additional serving clusters

• no proprietary storage formats

• no duplicated governance

Everything continues to use Delta Lake and Unity Catalog. Some will argue that this isn’t new technology, and that Databricks sees the growth of ClickHouse for example, and decided they needed to do something about that.

I agree, but I also think that Databricks providing this sort of extremely fast Compute option is ground breaking inside the Data Platform they provide. You simply CANNOT discount the reduction in complexity and code when you

Built for Operational Analytics

Databricks positions Lakehouse//RT for workloads that have traditionally been difficult to run directly from a data lake:

• customer-facing SaaS applications

• operational dashboards

• observability platforms

• security analytics

• embedded analytics

• AI agent retrieval

• interactive business intelligence

These are all scenarios where thousands of users—or AI agents—may issue queries simultaneously and expect responses in tens of milliseconds rather than seconds. It takes something tradtionally done outside Databricks, or at the minimum with third party tools, and brings it back inside the MotherShip.

All the chickens coming home to roost so to speak.

Simpler Operations

Lakehouse//RT also introduces a different compute model. Instead of selecting warehouse sizes manually, Databricks automatically determines the appropriate baseline compute. Rather than scaling by duplicating entire warehouse clusters, it incrementally adds or removes nodes as concurrency changes, aiming to improve utilization while reducing costs.

Governance Doesn’t Change

One of the more compelling aspects of Lakehouse//RT is that governance remains centralized. Since the data never leaves the Lakehouse:

• Unity Catalog permissions stay intact

• security policies are defined once

• business logic isn’t duplicated

• data lineage remains consistent

Organizations no longer need to recreate governance rules inside a separate serving database. Governance is indeed the next big topic in the world of data and AI. Security breaches every work, Claude 100X Engineers releasing buggy code left and right.

Can’t be too careful these days.

How It Fits Into Databricks’ Bigger Picture

Lakehouse//RT makes even more sense when viewed alongside Databricks’ other announcements this year.

• Lakebase provides a PostgreSQL-compatible operational database.

• LTAP unifies transactional and analytical storage on a single copy of data.

• Lakehouse//RT provides millisecond analytical serving directly from that same data.

Together, Databricks is attempting to collapse what has historically been three separate systems:

• OLTP databases

• analytical warehouses

• real-time serving databases

into a single platform built around Delta Lake, Unity Catalog, and specialized execution engines.

My Take on RT

This feels like one of the most strategically important announcements from this year’s Data + AI Summit. The performance numbers are certainly impressive, but the bigger story is architectural simplification.

For decades, data teams have accepted that customer-facing applications require a separate serving database. Lakehouse//RT challenges that assumption by making the lakehouse itself fast enough to serve those workloads. Will it catch? I don’t know. We will all find out in a year I guess.

The remaining question is whether those benchmark results translate to the wide variety of real-world production environments that rely on ClickHouse, Pinot, Druid, Elasticsearch, and similar systems today. If they do, Lakehouse//RT could remove an entire category of infrastructure from many modern data platforms.

Share

Lake Transactional/Analytical Processing Architecture

I’ve been waiting for this one since I was knee high to a grasshopper. Jezz, even Forbes is writing about this one, what the heck do they know about anything??

This is the one thing I wasn’t ready for, although it makes total sense. It was LTAP, short for Lake Transactional/Analytical Processing, and if you spend your days building data platforms instead of making keynote slides, this is the announcement that deserves your attention.

The elevator pitch sounds almost suspiciously simple. Databricks wants operational databases and analytical workloads to operate on the same copy of data, eliminating CDC pipelines, ETL jobs, replicas, synchronization processes, and the collection of brittle plumbing that has somehow become accepted as “modern data architecture.”

After reading both the press release and Ali Ghodsi’s interview explaining the thinking behind it, I don’t think this is really a story about ETL at all. It’s a story about removing a 40 year old architectural assumption that everyone stopped questioning years ago.

Today's architecture

Application
     │
 PostgreSQL
     │
 CDC / ETL
     │
 Data Lake
     │
 Analytics

For decades we’ve accepted that applications belong in one database while analytics belong somewhere else, usually connected together by a growing pile of Kafka topics, replication jobs, Airflow DAGs, managed CDC services, and a Slack channel dedicated entirely to asking why yesterday’s pipeline failed again.

The industry has spent years trying to make this architecture less painful instead of asking whether the architecture itself is the problem.

Databricks thinks it is.

The obvious comparison is HTAP, which promised to unify transactional and analytical workloads years ago. The problem was that HTAP largely tried to shove both workloads into the same engine, which meant eventually your dashboard and your checkout page were competing for the same resources.

Databricks is taking a different approach. Instead of building one engine that tries to do everything, they’re building one storage layer that multiple specialized engines can operate against.

LTAP

          Delta / Iceberg
                │
   ┌────────────┼────────────┐
   │            │            │
Lakebase   Lakehouse   Lakehouse//RT
  OLTP        OLAP        Real Time

That distinction is important because LTAP isn’t replacing PostgreSQL with Spark, and it isn’t asking Spark to become an operational database. It’s saying the engines can stay specialized while the storage becomes unified. That is a much bigger architectural shift than “we removed ETL.”

• The other piece that clicked for me came from Ghodsi’s explanation of why they’re doing this now. His argument is that the real pressure isn’t coming from human developers anymore. It’s coming from AI agents. Humans create applications relatively slowly. Agents don’t.

They create databases, clone environments, test ideas, throw them away, and repeat the process constantly. Databricks claims roughly 80 percent of the databases on Lakebase are already being created by agents rather than people.

Whether that number surprises you or makes you instinctively reach for a fact check, the direction is hard to argue with. Infrastructure built around dozens of data copies and endless synchronization simply doesn’t scale when your primary users become software instead of humans.

The real takeaway here isn’t that Databricks found a clever way to remove another Airflow DAG. They’re trying to remove entire categories of infrastructure. No operational database replica feeding analytics. No warehouse copy that’s always fifteen minutes behind production. No “Zero ETL” marketing that quietly hides another synchronization service behind the curtain.

• One copy of the data, multiple engines reading and writing it, and governance living in one place.

Will it work? That’s the billion dollar question, or perhaps the hundred billion dollar question given Databricks’ valuation. Lakebase still has to prove it can be a production operational database at massive scale, and LTAP has to demonstrate that this elegant architecture survives contact with messy enterprise workloads.

But if Databricks pulls it off, we may eventually look back at maintaining separate OLTP and OLAP systems the same way we now look at nightly FTP jobs and hand-written shell scripts. Necessary once, painful always, and eventually replaced by something that made us wonder why we tolerated the old way for so long.

We discuss Ethan’s journey into tech and data engineering, the evolution of open table formats like Delta Lake and Apache Iceberg, and what it actually takes to build scalable enterprise data platforms in highly regulated environments like big pharma.

We also dive into:

• delta-rs and the future of Delta Lake outside Spark

• Lakehouse architecture and open catalogs

• Rust in the modern data ecosystem

• Data platform governance and scalability

• Enterprise analytics and infrastructure

• The future of agentic analytics and AI-enabled data systems

• Lessons learned building large-scale data platforms

If you’re interested in modern data engineering, open source infrastructure, lakehouses, or the future of analytics engineering, this is a great conversation.

I’m going to tell your mom. I’m going to do it. You can’t stop me. And you, Grandma, for that matter. I’m calling them right now, like now. You’re an addict; you have to stop. Someone has to stop you. Might as well be me.

It started out harmless, didn’t it?

But now things have changed, you’re hooked on The Vibes. They got their claws in you. Look at yourself. You should be ashamed. Sitting high on your Macbook Tower all these years, God’s programming gift to humankind, those LLMs came along, you were enamored, Gastown and all, those tasty token morsels.

More tokens, bro, just a few more tokens, and you will finally become Neo and enter the Matrix.

Ok, on a serious note, if I have one in my body, what’s the big deal? Why are the token addicts crying foul and howling at the moon?

• Well, the signs are there; people from Fortune to LinkedIn are complaining about it.

What I’m still trying to figure out is how the collective “we” didn’t see this coming. Is there anything more classic in the marketing of drugs and AI, since the beginning of time, than to get ‘em hooked and then do the old bait and switch on pricing?

Heck, I don’t know if the pricing increases even matter. I mean, if the C-Suite wants to invest in AI, then who really knows whether the whole pricing strategy and increase are a big deal? It’s hard to turn the ship around once it has sailed.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

If you live under a rock and don’t know what I’m talking about, basically …

The era of heavily subsidized, flat-rate AI pricing has ended as both GitHub Copilot 
and Anthropic transition to token-based or usage-based billing. 
Users are facing significant bill increases and credit depletion due to the
 high token cost of long, autonomous coding tasks.

There are probably multiple factors behind the “why” folk are having second thoughts about token prices.

• Cost

• Risk (general)

• Lock-In

• The future

• Good judgement

• Privacy

• Freedom

The same leaders who’ve been proud of escaping “vendor lock-in” and other sorts of evils are the ones who recently woke up to the news that their dev token costs/bills might go through the roof, and are now crying wolf. Ironic.

I mean, hindsight is 20-20, but it wasn’t rocket science to figure out that something funny was going to happen at some point. Once you have the masses drinking from the AI teats, money talks ya know.

Also, I’m saying you have to abandon, or should abandon, tools like CoPilot or Claude Code because you’re scared, and overreact. There are many obvious, viable ways to reduce token usage that require little to no effort.

• Review CLAUDE.MD files and context

• Implement tools like Caveman

• Get better at prompting

• Adjust context

The truth is, I have great faith in the indelible human spirit. We find solutions to most problems, including AI token maxxing. Everyone has been sloppy because we were allowed to be. If we “have to” be more judicious, we can be.

Open-source alternatives to break free from token prison.

All that being said, in the classic white hat hacker, open source spirit that never dies … there is, and has been for a while, a strong undercurrent to have total control, total free control. The question is, can it be reasonably achieved?

• Can we find open-source alternatives we can run locally on our machines that deliver reasonable output and performance?

Most people are not going to go buy a Mac mini just to run a model for themselves. They will just pay the money to the SaaS Lords and move on. There are also other important questions to ask.

• Once we’re used to Claude Code and Anthropic speeds (from prompt to result), can we achieve them locally with any setup, or anything even close?

• We used to handroll code a few years ago. Can we be patient enough to wait 30 seconds to a minute for a response?

Something tells me that in the Instagram and Amazon life we live, once we’ve tasted that token fruit upon our digital tongues, it’s hard to find something else “good enough,” when it actually is good enough.

Will the code or systems design output meet our expectations? Will it take 30 seconds longer than we want, making it feel like years by comparison? Is the setup and installation overly burdensome?

As much as I would like to believe the best, the truth is humans are fairly predictable.

Subscribe now

Starting with OpenCode.

So, let’s start our possibly long and forelorn journey down the winding road of token freedom. Like Beowulf of old, we seek new lands and are ready to battle new monsters. I don’t expect this adventure to be free from heartache, but I’m sure we will learn something along the way.

• First things first.

I’m going to simplify my approach to this and break it up into two different logical pieces.

Think about it like your Claude Code or GitHub CoPilot setup. You have some “agentic tool” on your machine that you probably interact with. Then, you are using some remote LLM model via some API from Anthropic, OpenAI, whatever… these two pieces combined allow you to Gastown your way to glory.

You can use what you want, but let’s pick the cream of the crop in open source, that is, OpenCode.

So, let’s get to it. Easy enough. Find installation instructions for yourself here.

The rest is smooth enough, you’ll probably wonder why you didn’t do this a year ago.

Ok, so this is kind of deceptive up front. You’ve only fought half the battle at this point. By default, OpenCode will just look at your environment and find whatever default model it can find that you’re probably already using, things like OPENAI_API_KEY, ANTHROPIC_API_KEY.

Sure, we are now using an open-source coding agent/tool, but if we are trying to break free from our token addictions… then we still need to find a small, runnable, local model to wire into OpenCode.

If you want to understand more about using different LLMs with OpenCode, you can read their docs on the subject here.

Share

Small language models for coding tasks.

Enter the Rabbit Hole. What hole? The hole of the endless Reddit threads on what is the best “small language model” you can use locally for decent results. This is where personality traits and life outlook come into play.

Everyone works on different tasks, cares about different things, and will find some models better than others for various reasons.

I’m not here to jump into this debate on which SLM is the best for coding. It changes all the time and will continue to change, hopefully getting better and better as time goes on. It’s simply hard to compete with companies like OpenAI or Anthropic, backed by the Deep State, Bigfoot, Aliens, and billions of dollars.

So, I want to find something that is “good enough” in basic coding tasks. The idea being, in the real world, we could mix this setup with our Token Masters, say start out with OpenCode and some SLM, get the grunt work done, and maybe fine-tune with OpenAI or Anthropic.

• Back to the problem at hand, let’s pick an SLM and get it hooked up to our local OpenCode.

Ok, so in a somewhat satirical twist of fate, the once “don’t be evil” Google, according to the internet, is going to save our bacon on this one.

It just so happens those little blighters fine-tuned a coding-specific version.

So, I think we just give this CodeGemma 7B a try, wire it up to OpenCode, and let ‘er rip.

This might seem a little strange, but the easiest way to get CodeGemma onto our local machine and running is to use Ollama. I’ve used Ollama plenty in the past. It’s easy to get.

ollama run codegemma

So we have CodeGemma on our machine, thanks to Ollama. Don’t you love open source? Now we can hopefully configure our OpenCode to run CodeGemma and see whether we shed tears of joy or sorrow.

• Next, we need a little JSON config magic to point our OpenCode to our Ollama Gemma model. Mouthfull.

Next, we do a little vim’ing.

>> ~/.config/opencode/opencode.json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "codegemma:7b": {
          "name": "CodeGemma 7B"
        }
      }
    }
  },
  "model": "ollama/codegemma:7b"
}

Now we can double-check what model OpenCode is using.

opencode
/model

Doing a thing.

Well, we've done what we set out to do today. Who do you feel? Freedom? Anarchy? Rebel? Rich? Wow, a little bit of work and swimming against the stream, we have unshackled ourselves from the token mongers. Better sleep with one eye open tonight.

• I mean, the real question is … will it perform?

Beauty is in the eye of the beholder. I have no idea; maybe it will be too slow or just produce horrible output. Maybe it can’t do specific Data Engineering tasks. I don’t know.

Let’s just do something simplistic.

Have it read an open-source Divvy Bike Trip CSV file, and use maybe DuckDB or Polars to do some analytics, maybe a simple groupBy and aggregate.

Here we go.

So, we are learning something: one, we can fail immediately; two, apparently, all free open-source models are not created equal. I’m no savant of running local models; apparently, this GemmaCode doesn’t support tool calling, aka doesn’t expose the methods that OpenCode would need for integration.

Maybe that is something we could figure out and tweak, but look, I’m approaching this from a caveman perspective. I want an easy setup that anyone can tackle.

• Anywho, I am cornfeed and Midwest-raised, and I don’t quit easily. Onto the next model. Qwen, my love.

ollama pull qwen2.5-coder:7b-instruct

Of course, we need to update that config file.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "qwen2.5-coder:7b-instruct": {
          "name": "Qwen 2.5 Coder 7B"
        }
      }
    }
  },
  "model": "ollama/qwen2.5-coder:7b-instruct"
}

Ok, let’s try that data pipeline again with Qwen.

Make a Python file that uses DuckDB to read the CSV file 
202605-divvy-tripdata.csv, count the number of trips per day, 
write results to another CSV file. So count the
 column ride_id and group by started_at cast to a date.

Then I left and had supper with the family. Why not. Isn’t that the point of Agentic Coding? You just tell the Agent to do a thing, leave and do other things, then come back later to check on that thing?

All I know is when I got back from supper, it still wasn’t done. Ain’t not Anthropic, told you so. Look, that little stinker is burning 30% of my CPU.

I mean, I’m using what I consider a pretty standard MacBook.

What in Thor’s Beard? 33 Minutes for that task. That ain’t going to work, but I guess if you’re like Tiny Tim peddling the streets for tokens, time is of little importance.

• Also, what do I know about tools and OpenCode? It didn’t actually write the file. I just spit the Python out.

Before I start complaining about how crap this all is, let’s see if this code actually works. We will put it into a Python file ourselves and run it. Clearly, it’s tool calling ability, you can see it trying to write a Python file … is a little wonky, maybe stuff you can get working?

Off the bat, I can see its Python skills are a little sub-par.

What’s that saying your Grandma was always carrying on about? You get what you pay for??

• SQL query has wrong “ vs ‘

• df column referenced incorrectly, needs [] in a few spots

Fixed code, by myself, that is, looks like this and runs great, outputs as I asked.

Results.

Ok, so clearly, while my local model is mostly, well, only using CPU to solve these problems, this thing is going to crawl. Now you know why crypto bros have been ordering GPUs for years now. Also, you now know why things like Claude Code, combined with some Anthropic model behind an API, are so compelling, and also why those companies…

• Hire smart people

• Burn a lot of cash

• Suck up all our groundwater into data centers

• Use all our electricity

Takes some of that there “gumption” like we call it around here, for you to send some massive context over the wire and across the country, and get back over that same wire … a correct answer.

Subscribe now

So … yeah …

To be honest, I’m running out of gumption. Don’t tell anyone. Just give me my basket of nice hot tokens and let me snack on Claude Code. If you want me to reduce token costs, I will just sprinkle some caveman on it, improve my prompts, and cut out all unnecessary context that is hidden around here and there.

Give em’ the money, give me my tokens.

See, what did I tell you? I figured we could get something working, break free from the rest of the lemmings running over the AI cliff. Pretend we are rebels and run our own local model.

Life isn’t a movie; you haven’t figured that out yet? No free lunch? Never heard of that either?

I’m sure there are 16 things I did wrong that I'll be told about; the Reddit rabble will probably figure it out. All you have to do is meet them under the old oak tree at midnight and sign your name in blood with a stick. Who knows.

• More hardware, more GPU, better configs, I don’t know.

Just Google it a little, and you will see what I mean. There are already a million YouTube videos and other articles telling you they have cracked the code for the perfect, snappy model that performs best.

Good luck.

This is something I want to return to, maybe in a Part 2, do a little experimenting and research, see if we can find a model that gives good results on a decent-sized machine like this. Maybe it isn’t possible yet, who knows.

Yeah, I know we can solve lots of compute problems with more RAM, CPU, GPU … but not only did I want to see if it was/is possible to break free from the token tax, which, yes, it is possible, but what is it like for the average Dev who wants to do that?

It appears not to be THAT easy. Yeah, I didn’t spend much time on it, or trying to solve it yet, that can come later, I just wanted to dip my toes in the water and see what the world had in store for me.

Leave a comment

In this conversation, we go deep into the reality behind the AI hype, what makes engineers valuable (it’s not writing more code), and why the future of the field looks very different from what most people think.

We talk about the shift from coding to system thinking, why fundamentals matter more in the age of AI, and how junior engineers will need to evolve as tools like Claude and ChatGPT take over the “grunt work.”

Victor also shares hard-earned lessons from teaching, startups, consulting, and building systems at AWS, along with practical advice for engineers looking to stand out in a crowded, uncertain job market.

This is not a hype conversation. It’s a real look at where things are going.

🔑 What We Cover

• Why AI is making fundamentals more important, not less

• The biggest mistake engineers make is chasing promotions

• How to actually become a high-impact engineer

• Why does doing more Jira tickets not matter

• What’s broken about today’s interview process

• The future of junior engineers in an AI world

• Tactical vs strategic engineering (and why it matters)

• Why most AI-generated code is still “low quality.”

• How to think about career growth in a weird job market

💡 Key Takeaway

The best engineers aren’t the ones writing the most code—they’re the ones who understand systems, think long-term, and can drive decisions.

Ok, sometimes I honestly get amazed at what squeaky wheel gets the grease and what doesn’t in the data community at large. *Sigh. Well, I sort of get it, but as someone who loves the Single Node Rebellion, it’s hard not to call my mom about this.

The world, and I, have embraced the Lake House as the data platform architecture of choice (yes, it is an architecture), Delta Lake and Iceberg becoming one, we have all drunk of the chalice and are reluctant to put it down.

Part of the problem, if you can even call it that, is that we (the collective we) have, as a side effect, become addicted to clusters and high-cost compute. Sure, Databricks Serverless has come along, and is indeed welcome; tools like DuckDB, Polars, and Daft have for some time supported Unity Catalog access to the Lake House.

• But we don’t turn a blind eye to the complexity and issues this might create.

In the past years, I’ve been the dude pushing DuckDB to prod inside Lambdas and letting ‘er rip on Unity Catalog Delta Lake tables. But ya gotta be careful when you're trying to touch the sun. At the most basic level, Delta Lake and like Lake House storage layers are all about transactions, and once that word appears, it’s easy to blow stuff up.

How do you ensure you’re working on the right table version? How do you allow multiple writers, say DuckDB, to hit the same table and not FUBAR your production tables?

That’s what we are going to talk about today.

Le Problème (the problem)

I sorta mentioned the current “problem” with the Lake House architectures for the last few years. They have mostly been one-trick ponies, wonderful in their inception and in delivering data at scale, but poor at adopting real-world multi-tool read/write support.

Sure, we’ve had read access for a few years now for a variety of tools, but that only gets a guy so far. The ever reached for golden star that unlocks the potential for bespoke, handcrafted, multi-engine unlimited access and interaction has been mostly a untouched relic used by delta-rs and dynamodb mythical creatures.

Only a chosen few have the courage to walk down that path less traveled.

So, for the masses, we’ve had to stick to our Spark Clusters, Serverless, Databricks Connect, or whatever … to get the true read, and more importantly, write semantics, needed to build most production pipelines.

Sure, we’ve had DuckDB and Polars integrations into Unity Catalog for the brave, but you were still playing with fire and banging rocks together.

What were we missing?

Simplistic concurrent writes via common data tooling.

Simple is good, doesn’t have to be fancy, to open up the Lake House architecture to be more …

• flexible

• multi-engine

• full production-like support for all operations

Is that so much to ask for?

Then the Lord said … “Let them bring forth Catalog Commits.”

Let’s get to the matter at hand. Also, just because I’m singing the praises of Unity Catalog and Catalog Commits here, I will make some critical comments at some point. Ya’ know I bring the heat at all times and in all seasons.

I’m not going to give a ton of background here, just the basics, so everyone is on the same page about why Catalog Commits are a big deal. At the core of the problem, Delta Lake provides ACID capabilities on top of object storage. This makes the implementation simple, and the need for a Data Catalog simplifies the architecture.

But also surfaces issues. Databricks sums them up well.

• “… external engines writing to Delta tables directly in object storage cause catalog metadata, like schemas, to silently diverge from the actual table state.”

• “… every engine, tool, and agent can access tables differently, resulting in fragmented table discovery, inconsistent auditing, and no standardized enforcement of row or column-level controls across systems.”

• “… open lakehouse architectures historically have not supported atomic writes spanning multiple tables.”

The only real option to solve these issues was/is for Delta Lake to follow in Iceberg’s footsteps, for once, and move towards making a Data Catalog the primary means by which someone or something interacts with the Lake House.

Again, why is this a big deal? Well, because “we” have a whole new conceptual data-processing and pipeline architecture available now … for pretty much any and all Lake House ingestion and processing.

• Something we didn’t really have, or didn’t have much of before.

I think I’ve been pontificating enough at this point, and I would like to simply show you, on my personal AWS and Databricks accounts, how we can think about changing the data processing paradigm in the Lake House now.

• Simplicity, don’t forget that part. What I’m most excited about is the simplicity of this architecture.

Using DuckDB + AWS Lambdas + Databricks Delta Lake

Ok, so there is most likely no more basic and common data ingestion pattern on a Databricks Lake House platform than the simple ingestion of CSV files into some Medallion Architecture.

Typically, this would be done with a Databricks Job, using Spark. Not a big deal, except for your compute bill at the end of the month, AND the simple fact that even processing a few hundred CSVs every day is simply a little Spark overkill. You don’t exactly need distributed compute to do that.

So, what if we could just have some AWS Lambdas with triggers watching some S3 bucket(s) where the CSV files land, and let DuckDB inside the Lambdas write that data ingestion into the Lake House concurrently and at scale, without any worries?

Talk about simple. And fun.

First, we need a Catalog Managed Delta Table.

( all code on GitHub )

So, I’ve already got some Delta Lake tables banging around in my Databricks account. We will use the old and trusty Divvy Bike trips table and use some of those open-source datasets/CSVs for our little project.

First, let’s run a command to upgrade our nasty old Delta Lake so it can handle new-fangled stuff.

Next, we need two things related to our Catalog: either OSS Unity or Databricks Unity.

• Workspace URL

• PAT token

This is how our DuckDB, or whatever engine for you, will hit our Lake House from inside an AWS Lambda.

DATABRICKS_HOST = os.environ["DATABRICKS_HOST"]
DATABRICKS_TOKEN = os.environ["DATABRICKS_TOKEN"]

Also, let’s use a Dockerfile we can build and an ECR with AWS for running the Lambda.

FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.12

RUN pip install --no-cache-dir uv && \
    uv pip install --system --no-cache "duckdb"

RUN mkdir -p /opt/duckdb_extensions && \
    python -c "import duckdb; c=duckdb.connect(config={'extension_directory':'/opt/duckdb_extensions'}); c.execute('INSTALL unity_catalog'); c.execute('INSTALL httpfs'); c.execute('INSTALL aws'); c.close()"

ARG DATABRICKS_HOST
ARG DATABRICKS_TOKEN

ENV DATABRICKS_HOST=${DATABRICKS_HOST}
ENV DATABRICKS_TOKEN=${DATABRICKS_TOKEN}
ENV DUCKDB_EXT_DIR=/opt/duckdb_extensions

COPY main.py ${LAMBDA_TASK_ROOT}

CMD ["main.lambda_handler"]

Nothing much there, simple and simple. The lambda code itself is as uncomplicated as they come. These are the sort of things we should all default to.

import os
import shutil
import logging
import duckdb
from urllib.parse import unquote_plus

logger = logging.getLogger()
logger.setLevel(logging.INFO)

DATABRICKS_HOST = os.environ["DATABRICKS_HOST"]
DATABRICKS_TOKEN = os.environ["DATABRICKS_TOKEN"]
_OPT_EXT_DIR = "/opt/duckdb_extensions"
DUCKDB_EXT_DIR = "/tmp/duckdb_extensions"

if not os.path.exists(DUCKDB_EXT_DIR):
    shutil.copytree(_OPT_EXT_DIR, DUCKDB_EXT_DIR)


def lambda_handler(event, context):
    record = event["Records"][0]
    bucket = record["s3"]["bucket"]["name"]
    key = unquote_plus(record["s3"]["object"]["key"])

    if not key.startswith("trips/"):
        logger.info("Skipping key outside trips/ prefix: %s", key)
        return {"statusCode": 200, "skipped": True}

    s3_path = f"s3://{bucket}/{key}"
    logger.info("Processing %s", s3_path)

    conn = duckdb.connect(config={"extension_directory": DUCKDB_EXT_DIR})
    conn.execute("FORCE INSTALL delta FROM core_nightly")
    conn.execute("LOAD delta")
    conn.execute("LOAD unity_catalog")
    conn.execute("LOAD httpfs")
    conn.execute("LOAD aws")

    conn.execute("CREATE SECRET (TYPE s3, PROVIDER credential_chain)")

    conn.execute(f"""
        CREATE SECRET (
            TYPE     unity_catalog,
            TOKEN    '{DATABRICKS_TOKEN}',
            ENDPOINT '{DATABRICKS_HOST}'
        )
    """)
    conn.execute("""
        ATTACH 'confessions' AS uc_catalog
            (TYPE unity_catalog, DEFAULT_SCHEMA 'default')
    """)
    rows = conn.execute(f"""
        INSERT INTO uc_catalog.default.trips
        SELECT * FROM read_csv('{s3_path}')
    """).rowcount
    conn.execute("COMMIT")
    conn.close()

    logger.info("Inserted %d rows from %s", rows, s3_path)
    return {"statusCode": 200, "rows_inserted": rows}

Then all we needed was the actual ECR image, the Lambda, the S3 Trigger, some files, and a sprinkle of testing. First, the ECR.

Once we have built and pushed the ECR image, we can get Lambda up and running.

After the S3 file trigger is attached to Lambda, it’s just pushing up some files.

Of course, you and I both knew it wasn’t going to work; come on, admit it. Hope springs eternal, they say. I did have my dreams set on this working, but this early in the game, I figured something would go wrong.

That’s just experience.

I piddled around trying to figure out what’s going on, but clearly, we got some tom-foolery going on in the Delta Kernel. Funny, it seemed to work fine on OSS Unity on both the DuckDB and OSS Delta blogs, but who knows what they were exactly doing that I’m not.

Anywho, you and I at least get the picture of what’s coming. I’m sure, based on what I know about the Delta folk and DuckDB, it shouldn’t be long before things get a little bit more firmed up and wrinkles worked out.

Yordan’s story starts like many of mine, early programming, gaming, PHP, Linux servers—but what makes this conversation interesting is how he evolved from a generalist software engineer into a data engineering leader without even realizing it at first.

We spend a lot of time digging into what actually matters in modern data engineering, and it’s not the tools.

We talk about:

• Why the industry went too far into complexity and is now swinging back toward simplicity

• The reality of running a data platform at scale (and why most teams waste time maintaining tools instead of delivering value)

• How to think about migrations the right way without breaking everything

• The difference between junior, mid, and senior engineers—and why ambiguity tolerance and impact matter more than coding ability

• Why “perfect” engineering is a trap and how to actually ship work that matters

We also get into AI, and Yordan has one of the more grounded takes you’ll hear right now. Most companies aren’t even close to ready for AI, and the idea that it’s replacing engineers anytime soon misses the bigger problem: messy data, unclear metrics, and weak foundations.

Check out Yordan’s Substack below!

Data Gibberish

Data Gibberish turns experienced data professionals into well-rounded leaders. This is how you stop being overlooked, work on the problems that matter and get paid what you deserve. Nobody else teaches you this in data.

By Yordan Ivanov

We also talk about:

• How AI is actually used on real teams today (not Twitter hype)

• Why juniors with AI can be risky without strong processes

• How to think about code reviews, testing, and slowing down when it matters

On top of that, we dig into content creation, Substack, and what it takes to stand out in a world full of generic AI-generated content. Yordan’s approach is simple: write from real experience or don’t write at all.

This is one of those conversations that cuts through a lot of noise and gets back to fundamentals, how to think, how to build, and how to grow as an engineer in a rapidly changing space.

I hadn’t thought about it much lately, but depending on your point of view, Kafka is either at the height of its rise or on a slow downward spiral. Maybe both? We do live in the age of abstraction; businesses at large seem to be less willing to pay hordes of Platform Engineers to babysit complex architecture.

• But, with the rise of the Vibe Engineer, solving your streaming problems is only one god prompt away.

The (streaming) complexity is seen as chink in the old armor by many a SaaS vendor, Databricks included. That’s how the world turns. Someone sees a “problem” and says, “Hey, I can do better. Come on over here, my friend, the water is fine.”

Can you blame ‘em?

Just like Spark getting it’s heels nipped at by annoying puppies (DuckDB, Daft, Polars, etc), the streaming world has seen its share of upstarts trying to make streaming easier and less complex.

Then along came a little birdie and whispered sweet nothings into my lonely ear, something mysterious and wonderful … words like … “Databricks … Streaming … gRPC … Delta Lake.” That was enough to pique my curiosity and make me want to find out more. Some people know my weakness(es).

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

I want to take a gander at Zerobus from Databricks, generally how we can think about it and compare it to streaming tech like Kafka, why it exists, and then the actual reality of trying to play with it.

• What is it

• How do you use it

• Real-life playtime

The truth often lies somewhere in between what we believe. Once you read the documentation and try things for yourself, you may find that things aren’t what they seem, or maybe they are.

You can’t know until you put your hand to the plow.

Share

Streaming + Lake House

So, we need to have a talk about overhead and complexity. But we can also talk about the truth. Streaming and near-realtime systems are becoming increasingly mainstream in data culture. It also goes without saying that, in the current economic and business climate, not everyone is willing to overlook the costs and labor intensity of infrastructure or long-running streaming clusters.

One can argue that Kafka is going nowhere, and that is true. That’s like saying DuckDB or Polars will kill Spark. No, they are not. They just eat at the edges. This is the relationship of Zerobus to Kafka. At least for now.

The fact that Databricks probably spent an ungodly amount of time and money building Zerobus, when streaming has been around forever, and the market is full of offerings … tells you something.

• LakeHouse architecture is here to stay.

  • It’s the data layer of choice for modern orgs

• Streaming to a LakeHouse is problematic.

  • Zerobus makes it easy?

With all that as a way of introduction, let’s dive in.

What is Zerobus?

This is an interesting take on streaming data, eh? If you come from a world where the streaming platform is handled by a legion of engineers, or a few zealots, where most of the time is spent tuning, configuring, dealing, maintenance, upgrades, etc.

Well, then this probably sounds like black magic. One would assume, in Databricks fashion, that is the point. It’s rare the Databricks does something at %50 effort, or just does the next boring thing.

You can expect Databricks to introduce any new product or feature that tries to flip the script and become a serious contender in whatever space it enters.

• No doubt there are plenty of large enterprise customers of Databricks who did nothing but complain about the complexity and brittleness of large-scale streaming into their Lake House (probably a Delta-style lakehouse).

What better way to solve streaming in the context of a Lake House than…

• API interface

  • gRPC

  • REST

  • OpenTelemetry

• Serverless

• Push Only

Again, this is about the simplicity of ingesting streaming data directly into a Lake House, without the need for expensive, complex third-party tools to operate and maintain.

Yet another simplification of the Modern Data Stack.

Subscribe now

Learn by doing.

We could spend more time pontificating about the concepts and finer details of Zerobus, but I think it would be best to select an extremely simple use case, try it out, and discuss what we see as we go.

Before we begin, we need to identify just a few pieces of information.

• Our Zerobus Databricks Endpoint

• Create a target Delta Lake table.

• Set up Auth and permissions

• Choose Client SDK and write code.

Define your endpoint.

To get our Databricks Zerobus endpoint, you only need your Workspace ID and URL. Mine is something like this …

SERVER_ENDPOINT = "https://319592733000122.zerobus.us-west-2.cloud.databricks.com" # Workspace ID
DATABRICKS_WORKSPACE_URL = "https://dbc-9a64f31c-25b9.cloud.databricks.com/" # Workspace URL

Easy enough.

Define your service principal, OAuth secrets, and permissions.

Next, you will need, or should use, a service principal with secrets and proper permissions to said table. Maybe something like this delta-streaming

… OAuth Secrets and Permissions …

And now we have our OAuth thingys, we have everything we need to start writing the code.

Prep your Delta Table, or make one.

We will use the Divvy Bike trips open-source dataset, just because it’s easy and accessible. I’ve already set up a Delta Table for this dataset, so we'll use it as our target table.

Write the Zerobus Client.

Ok, so now we are at the point where we can actually get to using Zerobus, and they give you a few different SDK options to interact with Zerobus, depending on whether you are a fake data engineer, a power user like me, or maybe more of a real data engineer, like Scott.

• Oh, I almost forgot that dreaded pip install.

We are going to be combining Zerobus with PyArrow, aka Apache Arrow, because Arrow is the next big thing, you ninny.

Prep our Arrow schema.

schema = pa.schema([
    ("ride_id", pa.string()),
    ("rideable_type", pa.string()),
    ("started_at", pa.timestamp("ms")),
    ("ended_at", pa.timestamp("ms")),
    ("start_station_name", pa.string()),
    ("start_station_id", pa.string()),
    ("end_station_name", pa.string()),
    ("end_station_id", pa.string()),
    ("start_lat", pa.float64()),
    ("start_lng", pa.float64()),
    ("end_lat", pa.float64()),
    ("end_lng", pa.float64()),
    ("member_casual", pa.string()),
])

Anywho, next we write the rest of the code.

• Create Stream

• Read dataset

• Push batches to Stream

Y’all knew as well as me it wasn’t going to work the first time. I never lie to you, my fair-weathered friends. Call me Honest Abe.

I see no mention in the docs about not being able to use Serverless. Basically, I can’t even hit that Zerobus endpoint from my Notebook/Serverless. Not much to do besides attempt the same code on an All Purpose cluster and see what’s crack’en.

• Ah ha! So we get farther when we don’t use Serverless, maybe that’s not an option, and they don’t tell you.

Using an All-Purpose Cluster, we get an actual error about our Schema from our Zerobus SDK, so that’s a good sign.

I got any easy for this Sunny Jim. Just make all the fields STRING, we gotta break out our inner Junior Dev mindset here. Channel the vibes.

Dang, add a little Arrow batch.cast , and we are off to the races! It works!

Let’s take one of the 140,000 records we just streamed to our Delta Table with Zerobus and see if we can find it. Hot Dog, ain’t it a beauty?

Thinking about things.

There are a few things I’m mildly curious about. Namely, you have to imagine that the number of parquet files created will be a little crazy, depending on your settings, so you'd better manage that whole OPTIMIZE crap or your streaming batches.

• Also, just in case you think I’m lying to you, here is the History of that table. You can see the Zerobus ingestion there.

Here we have 8 files in this Delta Table after the streaming job, but before OPTIMIZE.

Afterward, that is OPTIMIZE, we are down to a single parquet file backing up the Delta Table, not 8. One can use one's imagination to consider how carefully one would have to manage real-time, high-volume streaming ingestion into a Lake House.

• Again, at the production scale, many hundreds of TBs or more, this would require some serious thinking.

Closing Comments.

I just want to spend a minute now that we’ve done a little dirty work, talking about what we’ve learned, and questions we might have about streaming to the Lake House with Zerobus from Databricks.

Firstly, we must admit two things that have been happening at large in the data world …

1. Streaming has become popular

2. The Lake House is the new data architecture of choice

Also, we would all have to admit that streaming is probably one of the most complicated and complex tasks one can tackle in Data Engineering. It’s a whole different animal than batch. Also, many companies and tools seem interested in simplifying streaming, bringing it to the masses, and lowering the barriers to entry.

Yeah, I know we just played with a toy example, but one has to admit, Zerobus is pretty slick.

• No big infrastructure to maintain and tune

• SDKs are simple and straightforward

• Integration into the Lake House is a breeze

Methinks that this streaming complexity problem, previously requiring bespoke and custom setups via Kafka and Spark Streaming were just a little much for some folks.

I mean, it does add a lot of complexity and breakpoints.

Anywho, now we have a new and wonderful option in Zerobus if we are interested in Lake House streams.

Based on our preliminary playing around, it appears to be a very serious contender, heck, they even have support for Arrow! That was a pleasant surprise.

If you have any experience streaming to a Lake House, drop a comment below and tell us about your setup!

Leave a comment

Jacob didn’t start in tech the way most people expect. He began in accounting, working with Excel and financial systems, before slowly realizing that the real problem he loved solving wasn’t finance, it was data pipelines. That path eventually led him deep into SQL Server, data warehousing, and ultimately to DuckDB, a tool that fundamentally changed how he thought about processing data.

Share

• What we get into is bigger than just tools, though.

We talk about why DuckDB exploded in popularity, what it gets right that traditional databases and even modern cloud warehouses struggle with, and why the industry may be swinging back toward simplicity after years of over-engineered “modern data stacks.”

There’s a really interesting thread here around how engineers accidentally created too much complexity, and now tools like DuckDB are winning by removing it.

We also go deep on the evolution of the data stack itself. From SQL Server’s “everything in one box” model, to the unbundled chaos of the modern stack, and now back toward a more unified, simpler approach. Jacob shares how MotherDuck is thinking about that shift and where things are headed next.

• One of the more important parts of this conversation is around AI.

There’s a strong argument here that AI doesn’t kill data engineering; it massively expands it. Instead of fewer queries being written, we may be heading toward a world where AI agents generate orders of magnitude more queries than humans ever could. That flips a lot of assumptions on their head, especially around things like data modeling, which suddenly becomes more important, not less.

Subscribe now

We also talk about:

• Why most Spark workloads are overkill

• When single-node tools like DuckDB actually win

• The real tradeoffs behind Lakehouse architectures

• Why data modeling is still critical in an AI-driven world

• How engineers should think about building in 2026 and beyond

This is one of those conversations that helps you zoom out and see where things are actually going, not just what tools are trending this week.

If you’re building data platforms, experimenting with AI, or just trying to simplify your stack, this one is worth your time.

If we had a dollar for every time someone came along to become the Apache Parquet killer, we would all be living on the side of a mountain tending to our alpacas. A boy can dream, can’t he?

Because it’s a holiday Monday, and I’ve been up since dawn running amok on the river, and now I’m lying on the porch with a cool breeze, it’s time we put our hand to the plow and turn up something interesting.

Today, that will be the Vortex file format.

At this point, half of you probably know more about Vortex than I. I’ve heard the name rattle around here and there, but never touched it. So, I don’t expect much today, really just a little poke here and there, a little benchmark to bring the craizes out of the swap, ya know, the usual stuff.

Vortex file format for ninnies.

It isn’t really apparent to me, on a simple poking around in the docs, what niche is being chased down by Vortex. I mean, if you read between the lines, and not hard to find lines, seeing worlds like “… vs Parquet” or “compressed columnar data,” heck, even a “unlike Arrow …” makes me sit up and take note.

• Seems like these little buggers are taking potshots at everyone worth anything today.

The GitHub page actually brings it home a little better.

“Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage.”
- GitHub

It also boasts some impressive features that stand out to me beyond what has already been mentioned.

• appears to be written in Rust

• “Zero-copy compatibility with Apache Arrow”

• Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more

• Modeled after Apache DataFusion’s extensible approach

So, we shall see, eh?

Python + Vortex and benchmarking performance.

Well, if any tool worth its salt wants to make a go of the data community, it must have first-class Python support; otherwise, all is for naught. Rustaceans might cringe, but that’s simply the way of the world.

Let us kill those two birds with one stone.

• Try out the Python integrations with DuckDB, Polars, and Datafusion.

• Benchmark those against each other, plus vs CSV, Parquet, maybe Lance?

We can see what it’s like to use Vortex inside Python code, and see if there is any indication its performance is a thing or not.

Remember, if you leave me nasty comments about scientific benchmarking, there is a high probability that I will make fun of you in front of 35,000 people who follow me on this platform. Just saying. Take your chances.

Next, let us grab some data to use. We will use the Backblaze open-source hard drive dataset about failure rates.

Let’s take these two zip files, each containing two quarters of data, totaling about 23.91 GB on disk.

• This should be enough for our purposes.

First, let’s run a simple DuckDB query on these raw CSV files to get a baseline, then we will convert to Vortex and test again.

• All this code is on GitHub, you hobbit.

Total failure days: 181
Runtime: 25.465s

Ok, longer than I thought. Let’s run the exact same code on raw CSV with both Polars and DataFusion before we switch to Vortex, and then test Parquet and Lance as well.

Oh … what do you flipping know, another one of those mysterious Polars failures that doesn’t exist, and you are anathema and a heretic for even saying it.

I highly suggest you go read this. And also, I encourage you to go to Google OOM Polars issues yourself, don’t let that powerful and slinking Nazgul scare you away, you will find enough evidence yourself. You can only hide the pea for so long.

• This is why I had to rip Polars out of production: simply unpredictable and unreliable in an unacceptable way, for tasks that other tools handle with ease.

Anyway, stick Polars in the ditch, let’s move on to DataFusion.

Oh dang, Datafusion is fast.

Total failure days: 181
Runtime: 5.106s

Ok, let’s move to Parquet files.

So, let’s convert all the CSV files to Parquet and re-run those scripts with DuckDB, Polars, and Datafusion on them. See what’s cracken.

• 184 CSVs → 20 Parquet files (10 per quarter, last batch 2 files each).

Again, all the code is on GitHub, and we converted all our previous scripts to just read Parquet instead of raw CSV. You can go look at the scripts if you please. I’m just going to show results here.

  - failures_by_day_duckdb.py — 0.125s
  - failures_by_day_datafusion.py — 0.370s
  - failures_by_day_polars.py — 0.193s

Well, that is quite the difference, eh.

Ok, let’s move to Vortex files.

So, let’s finally get to what we’ve been waiting for this whole time: will these packages really integrate well with Vortex, and will the performance be noticeably faster than Parquet?

• First, let’s convert from Parquet to Vortex with DuckDB.

With that done, let’s run the benchmarks with the same set of tools again for DuckDB, Polars, and Datafusion on Vortex. I will put the DuckDB example in; you can check the rest in GitHub, and print the results only here.

  - vortex_scripts/failures_by_day_vortex.py — pure vortex scan with filter pushdown, 0.111s
  - vortex_scripts/failures_by_day_duckdb.py — DuckDB querying VortexDataset (PyArrow interface), Runtime: 0.201s
  - vortex_scripts/failures_by_day_polars.py — VortexFile.to_polars() LazyFrame, Runtime: 0.114s

FYI, the integrations aren't as solid as they claim. The DuckDB vortex extension blows up with memory errors. OOM like Polars on CSV files.

Polars only seems to support per-file-type code; I couldn't get any globbing patterns to work. On top of that, we convert to PyArrow and Polars right away in that code.

Heck, maybe I’m doing stuff wrong, but the farther I get into the weeds with Vortex, the more I realize this might be early days yet. Seems there are still some problems to solve and wrinkles to iron out in the Python ecosystem.

• We could probably make DuckDB work with Vortex by doing what we did with Polars, convert to Arrow right away?

Blah, performance is fine, but that is some nasty-looking code. No fault of these third-party tools, just a little early in the lifecycle methinks.

Anywho, here ya go.

I didn’t learn a ton. Maybe the lift over Parquet is larger on massive datasets, but I'm not sure it’s worth the hassle of subpar Python framework integrations and ugly code, along with OOM issues when you try to read directories of Vortex files.

You’ve seen it before. Don’t lie, I know you have. Java has the Stream API. .NET has LINQ. We’ve even talked about lazy evaluation ourselves.

When we last looked at lazy evaluation, it was in the context of doing work only when it’s needed, rather than preemptively doing all the work and risking that much of it wasn’t needed. YAGNI.

All that said, these streaming models expose other benefits. And I, your friendly neighborhood Anonymous Rust Dev, am here to tease some of them out.

Declarative code

First, a shameless plug for our recent conversation around declarative pipelines. In it, Dan looks at Spark and the SDP framework with examples.

SDP isn’t the first time declarative code has appeared on the scene. In fact, traces of it can be found back in the coding antiquity — ML originated in the early 1970s and is the precursor to many modern functional languages like Haskell or Scala. And I know my audience; SQL’s SELECT It is another popular declarative grammar.

It’s understandable how imperative became the lingua franca of the programming world. For instance, just looking at C, one of its goals is to closely mirror what the underlying hardware is doing, and if you’ve ever cracked open assembly code, you’ll see imperative code at its most fundamental level. For those who don’t know assembly, I don’t plan to show you any code today, but it’s literally just a sequence of atomic steps like “add this to that” or “go to this location in code”.

• Procedural code is easier to understand when your scope is small. It’s very easy to reason about an addition statement. But when looking at large-scale problems, many tiny instructions start to turn into noise. Naturally, you can (and should) refactor your code to hide some of that. In a perfect world, if you’ve done a good job of refactoring and structuring your codebase, the code should flow like prose — or, to put it another way, it should tell a clear story.

And that is where declarative programming comes in. Baked into its entire premise is the notion that, rather than giving you a series of do-this-then-do-that statements, it instead models the flow of an application, and more closely represents many problem domains.

Functional programming

The “functional” in the name isn’t talking about whether or not it works, but rather the unit of work being employed. If this video by Brian Will doesn’t quickly sour you on OOP, I’m not sure how long you’ll be able to stick with me; the core premise I take from it is that OOP doesn’t really model how things work in the real world. Conversely, if you’ve ever seen a flowchart and understood it, you intuitively know functional programming.

Don’t get me wrong, OOP has its place, but it’s overused and employed for the wrong problems. If you’ve ever spent time in old-school Java and C#, you understand this better than most — everything is derived from Object, and if you want to execute behavior in your code, the recipe is:

• Have a thing

• Have the thing do something to itself via methods

Sounds passive to me. All of your code starts on the premise that it’s attached to a thing, and at some point, the “you must have a root thing to do all the stuff” narrative became so tiring and tedious that C# introduced syntactic sugar to alleviate it with their “top-level statements” concept. Now, you don’t need a bunch of namespace and class boilerplate to start doing stuff — it’s still there, lurking behind the scenes, but programs start to feel more like a series of actions and less like a bunch of objects that are being forced to stand in for behavior.

When functions are first-class citizens, behaviors are easier to express... and arguably, to understand. Where OOP inevitably drives you to use class structures and inheritance models to contain, protect, and manage application state, functional code tends to focus on “having data” and a means to transform or process it.

Now, that recipe becomes:

Do stuff

In practice

Relax, I’m not trying to convince everyone here to become a Scala dev. While not as elegant, Python has some constructs that make this somewhat ergonomic.

First, I’ll introduce you to the notion of a “point-free” or tacit style of programming. That Wikipedia link offers a Python example that illustrates how to “compose” a series of functions into a wrapping function.

This should intuitively make some sense when you see it. If you’ve ever baked a cake, you understand the black-box idea of “baking a cake,” even without memorizing the individual steps. Of course, you realize there are in fact several atomic steps that must be executed (e.g., procure eggs, store/refrigerate eggs, remove eggs from storage, break eggs, blend eggs, etc.), but many of those details are composed of parent steps that themselves are composed into the recipe as a whole.

Functional programming follows the same spirit. I’m switching to TypeScript for this illustration, since it does a good job of being readable while still giving us type descriptions:

interface Ingredient {
    name: string,
    quantity: number,
    units: 'each' | 'grams' | 'milliliters',
}

type Cake = unknown;  // We'll figure this out some other time

function bakeCake(ingredients: Array<Ingredient>): Cake {
    // TODO
}

First, you can see I “baked” in some assumptions (sorry, couldn’t help myself) — namely, that you need ingredients as an input to produce the output Cake.

Want to see something cool? The function signature for a “good” (subjective measure) function should tell you something even without knowing the function or argument names: (Ingredient[]) -> Cake. You intuitively know, even without being told what the function’s name is, what recipe is being executed:

Scenario: (unnamed behavior happening here)
Given some ingredients
When we (insert function name here)
Then we produce a Cake

BDD (e.g., Cucumber’s Gherkin, as shown here) is an amazing way to test-drive systems, particularly those with an emergent design like the one we’re using here. We “know” there are some steps to producing a cake, and we can black-box that behavior in the meantime with a Scenario until we’ve had a chance to do some discovery and tease it out.

Returning to our bakeCake function, let’s try stubbing out some behavior:

function prepare(ingredients: Array<Ingredient>): Array<Ingredient> {
    let processedIngredients = [];
    // TODO: take input ingredients, do stuff do them, produce some "prepared" ingredients
    
    return processedIngredients;
}

type Oven = unknown; // Also figure this out at a later date

function preheatOven(tempCelsius: number): Oven {
    let oven;
    // TODO: do some stuff to make our oven hot here
    
    return oven;
}

interface Food { /* TBD */ }

// Fleshing out Cake just a bit more:
interface Cake extends Food { /* TBD */ }

function cook(ingredients: Array<Ingredient>, oven: Oven): Food {
    let result;
    
    // ???
    
    return result;
}

function bakeCake(ingredients: Array<Ingredient>): Cake {
    const preparedIngredients = prepare(ingredients);
    const preheatedOven = preheatOven(175.0);
    const cookedFood = cook(preparedIngredients, preheatedOven);
    
    return cookedFood as Cake;
}

Actually, if we’re being honest, the process of baking a cake doesn’t invent a new oven in the process; we’ve teased out a hidden requirement that makes me want to shuffle some stuff around to be more honest:

function preheat(tempCelsius: number, oven: Oven): Oven {
    // do something to our input oven to make it the right amount of hot
    return oven;
}

function bakeCake(ingredients: Array<Ingredient>, oven: Oven): Cake {
    const preparedIngredients = prepare(ingredients);
    const preheatedOven = preheat(175.0, oven);
    const cookedFood = cook(preparedIngredients, preheatedOven);

    return cookedFood as Cake;
}

While still sketchy on the details, we can now see the emerging design of a cake-baking workflow. Also, I have enough in place to refactor with composition in mind (relying on type inference where possible):

const bakeCake = (ingredients, oven) => cook(
    prepare(ingredients),
    preheat(175.0, oven)
) as Cake;

You’re free to have whatever opinions you like, insofar as to whether you prefer the refactored version or the more imperatively-styled variation that preceded it.

However, you have to admit, the relationship between input and output is clear enough at either point to describe it in high-level terms (e.g., via a flowchart):

Yeah, this is the first time we’re actually seeing the ingredients enumerated, but I couldn’t leave you hanging forever...

Subscribe now

Read more

Ben shares how he went from working in kitchens… to data engineering… to Facebook… and eventually walking away from it all to build his own consulting business.

And yeah, it wasn’t all glamorous.

“I was making the same money as Facebook… and I hated my life.”

We get into the stuff most people don’t talk about:

• What it’s actually like working in Big Tech

• Why high-paying jobs can still burn you out

• How he transitioned into consulting (and what people get wrong)

• The reality of modern data stacks and tool sprawl

• Whether data engineering is changing because of AI

• Why fundamentals still matter more than ever

We also go deep on where the industry is heading:

• Is the “modern data stack” breaking down?

• Are tools like DuckDB actually replacing warehouses?

• Is data modeling dead… or just not trendy anymore?

• What AI is really changing (and what it’s not)

If you’re trying to break into data, grow your career, or figure out where things are headed… this is one of the more honest conversations you’ll hear.

Ben also runs a course and community for those interested in getting into consulting.

I recently came across the most classic Data Engineering problem of the last 50 years. Duplicates records. It was some “old code”, if you call 5-year-old Databricks Spark code old, which is like 15 years in Databricks years. To me, it was the perfect example of taking time, a lesson in the most basic fundamentals of data.

It’s not a particularly earth-shaking problem in and of itself, rather boring, but it’s a story that re-enforces the age old ideals.

Since the days of my data youth, heck, harkening back to my LAMP stack years in college, data duplication has been an issue that somehow manages to creep into database tables of all shapes and sizes.

History is doomed to repeat itself; programming and logic errors, conundrums, and slipups are par for the course. I mean, those little blighter LLMS were literally trained on our collective sins, so what makes you think things will get any better?

That’s what I thought.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, 
and I believe it represents the future of Data Engineering. Content like this 
would not be possible without their support. Check out their website below.

We start at the beginning.

What is your life really like if at some point during the past 365 days you get a message from someone “on the business side” that so-and-so is showing duplicate records? A tale as old as time, this.

• First off, and before you throw a fit, hear me out: What is a duplicate record?

That is indeed NOT a stupid question. Didn’t your middle school teacher ever tell you that there is NO such thing as a stupid question? It’s true. When it comes to data, you simply cannot assume anything. Like anything … at all.

Sometimes duplicates are by design. It might have been a faulty assumption or a bad decision, but by design in many cases. One of the recurring fundamentals of data, when you encounter a new, or a new-to-you dataset, should be the question … “What is the grain of this dataset?”

Before running off to solve any duplicate data problem or looking for a bug that might not exist, one should start from the ground floor. It’s just good table manners, and good for you as a technical person.

It’s hard to solve a problem, especially a data problem involving duplicates, unless you have solved the many times tricky problem of figuring out what makes each record in the dataset unique. It’s never the same for any two datasets, or rarely is.

Also, before we talk about the problem I encountered, I think we should break down another great way to approach duplicate data to ease the stress of solving issues that can be buried in a complex and large system.

We should separate any data system or duplicate data research into three big boxes. These boxes can help us narrow down the scope of the problem and where to best unleash the Claude hounds to find it.

1. Source

2. Transformation

3. Destination

• This may seem painfully obvious, but when we approach a problem like duplicate data in large, complex systems, finding the bug can be overwhelming. Being able to logically and technically differentiate where a problem is, or isn’t, will make our jobs and time to resolution much quicker and less stressful.

Subscribe now

Back to the problem at hand.

So, back to my boring storing of trying to solve an intermittent duplicate data issue within a few short hours, so I could get back to the AI salt mines.

Here’s what I knew.

1. Duplicate data issue showing up inconsistently in a Web UI.

   1. Some days there are duplicates, some days are not.

2. Pipeline from source to finish included …

   1. Delta Lake House

   2. Databricks Spark transformation

   3. Python script to push said data to Postgres.

   4. Web App pulling data from Postgres.

Now, logically, the data problem could be anywhere, although the intermittent nature of the duplicate data would probably preclude problems in the Web App.

Also, when working in large data systems, the best you can do is control the controllables rather than blame others. My plan was simply to eliminate the possibility of duplicates insofar as the data systems I CONTROL.

A good rule of thumb for data, and life … control the controlables, don’t worry about the rest until you have too.

When I first heard the word “intermittent”, I had some suspicions in my mind. But, being the fundamentalist I am, I started with the Delta Lake Table.

Read more

In this episode, I sit down with Matthew Housley, a famed co-author of Data Engineering Fundamentals and former CTO of Ternary Data, to talk about what actually makes data teams successful and why so many organizations get it wrong despite having modern stacks, cloud platforms, and expensive dashboards.

• Matthew’s path is a little different than most. He started in academia as a mathematics instructor before moving into industry as a data scientist at Overstock.com, and eventually leading data strategy and analytics as a CTO. That mix of academic rigor and real-world execution gives him a very clear perspective on where things break down.

We get into the gap between data science and real business impact, why analytics foundations matter more than flashy models, and what companies consistently underestimate when building out data platforms. We also talk about what it actually looks like to transition from academia to industry, and how that shapes how you think about data problems at scale.

If you’ve ever felt like your data stack should be delivering more value than it is, this conversation will probably hit close to home.

⏱️ Topics we cover:

• Why most analytics efforts fail before they even start

• The difference between “doing data” and delivering value

• Data science vs data engineering vs analytics reality

• Academic thinking vs industry execution

• What CTOs actually care about when it comes to data

• Building foundations that don’t fall apart six months later

I like to take the middle road in life, not too hot, not too cold, right down the ol’ center lane. When something like AI comes to gobble us all up, I just smile and keep doing my thing. We live in a world now filled with three types of people.

• AI Gluttons

• Middle of the Road

• AI Deniers

You can guess which camp I'm sitting in.

Once the cat is out of the bag, it’s hard to get it back in. AI is here to stay; it will change the face of software forever, slow, fast, bubble, no bubble. It’s a changing.

So, if you have the luxury of being an AI Denier and never using it, well … good for you, although the next time you look for a job, you might get a surprise. I understand the argument about “Software as a Craft” and being the best at what you do. Such folk will always find a place.

But, indeed, AI is an innovation worthy of some use in the software world, me thinks.

Well, Deniers that have always been and always will be, and that leaves us with the AI Gluttons. Who knows who I’m talking about? It’s the same folk who, 5 years ago, thought their entire worth as someone who wrote software for a living was simply writing software.

• That was always a simplistic view of software and led (and still leads) to some of the worst teammates and code producers.

What do these new AI Gluttons do?

They say … give it to me, all baby, I want the whole thing. Every AI Agent, Gastown, every MPC server, and Claude Skill you can find on GitHub.

Read more

Typically, when the Harvard Business Review publishes something, especially techy, people tend to pay attention. Well, that is, unless it goes against the ultra-psyops-capitalism that drives most of the known world, in the form of extracting every useful drop of blood and life from the glassy-eyed masses that are too exhausted or addicted to the doom-scroll to look up for a minute.

Hey, am I just the kettle calling the pot black, while I sell you a 10-dollar-a-month subscription? Maybe.

Just when you thought the late-night working, on-call fearing, Slack notification twitching, deadline anxiety-ridden Software Engineer was at the end of his or her rope, one JIRA ticket away from giving it all up for chicken farming … the world in all its maniacal plotting dropped AI square into the face of every developer.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

All we have experienced and learned to this point has been nothing, just training, really.

It was just preparation to be plugged into the actual matrix, the Gas Town crazed C-suite and CTOs, whose eyes shine bright with endless Claude-driven possibilities of Agents upon Agents, calling each other in endless token burning loops, while investors pour bags of money on their already rich heads.

• Every normal path of the software engineering life cycle has been turned upside down, short-circuited, smashed, and thrown out the window.

In a blindly dizzy change of culture, the once clean-code-loving, JIRA-embracing, Agile acolytes have abandoned their first love for an Anthropic mistress, abandoning overnight 50 years’ worth of collective experience and knowledge, which was stripped away, sucked up, and fed into the darkness of Foundational Models as training data.

It wasn’t enough to simply turn a million programmers into token fools; once independent thinkers, we are all now sucking at the teat of subscription plans. We live in fear of being cut off from our new drug of choice, the token, the bringer of life and features.

The outcome and human cost of these Agentic Coding tools are as obvious as they come.

What it means to program in the Age of AI.

I feel I might have some insight here, having been on the giving and receiving end of the AI firehose of code, and now that we’ve been living in that world for a bit, it’s pretty obvious both the upsides and downsides we are dealing with in this Brave New World.

Look, my personal feelings on it all are irrelevant, as are yours. We are just little cogs in a big machine, washed along the flood of professional life, victims of the rains of the tech culture at large. You may tell yourself you sit aloof, you and your neovim, but the truth is you’re doing what you’re doing because you’ve been influenced, or pushed places by necessity.

• So what does it mean to deal with AI in all its reckless glory?

1. Senior+ Engineers have turned into glorified code reviewers.

2. It isn’t possible to keep up with the pace of AI output in a codebase.

3. You no longer understand the minutiae and details of an AI-generated codebase.

4. Your mind and soul are torn between doing the right thing (understanding and
 reviewing changes in depth) and meeting expectations.

5. Inexperienced devs and non-engineering stakeholders think they are smarter 
than they are (armed with Claude).

6. Bad designs and architecture are amplified.

7. You will get more “burned out” faster.

8. You let “things slide” that you normally would not have.

9. We have to deal with more organizational and professional chaos and uncertainty.

I think what it boils down to, and the end of the day, is the mental burden of the expectation that we use AI to move quickly and produce more, while still being held accountable for all negative outcomes and side effects of the software products we produce.

It is a classic problem: being held responsible for something you can’t control in its entirety.

A little interwebs search will assure you that this topic is on the back of the minds of a lot of folk, who know enough to be worried.

Why kick against the goad?

So what?

It’s all good and well to rage against the AI machine, but methinks that might be a fool’s errand. There probably isn’t much hope to make any gains against the flow of tech culture; it’s probably best to just find our way through it, if we can.

Now, more than ever, it’s important to do a few things to counteract AI burnout in your own life.

• Find hobbies and take time away from the computer.

  • Go outside, exercise, and read.

• Treat AI coding like another skill that’s important in the marketplace, don’t overemphasize it.

• Remember why you fell in love with coding, and do those things yourself regularly.

• Ignore the AI Doomers and the AI Groomers at the same time.

  • Take the middle road.

I think a healthy dose of reality in all its forms is a great antivenom for a mind and body plagued by AI burnout. For me, this burnout creeps in slowly and takes over before I know it, or realize it.

Context switching quickly, massive code changes, and PRs, the incredibly fast pace of development and new features, and worrying about quality, understandability, and best practices.

Slow yourself down.

While those around you move and ship at breakneck speeds, you should slow down a little. Take time to think through large design decisions, architecture, and systems.

Think critically and slowly about that AI-generated PR, just because it was spewed out in a day (what once would have taken a week), spend an extra day or two really understanding the business context, what’s happening, and hidden decisions being introduced.

Beat them at their own game.

Don’t try to escape the game; it’s been forced upon you, and it’s probably not going anywhere. Be optimistic in your attitude, remember why you do what you do, and why you love it.

They think AI will make them a 10x Engineer with little work and no sacrifice. You know better.

Go deep, understand problems like no one else. Ship at their pace when needed, and slow down when it counts. Take your time. Find joy in the great outdoors, and live a healthy lifestyle. The code will flow long after you are gone, and was flowing long before you arrived. Remember that.

Please leave a comment and let me know how you view the AI revolution and how you are dealing with it yourself.

Leave a comment

In this episode of the Data Engineering Central Podcast, I sit down with Neil Roberts — a developer who’s been through every major wave of the web, from BASIC on an Atari to modern TypeScript, and now deep into LLMs and agentic workflows.

This is not another surface-level “AI will change everything” conversation. We get into what is actually happening right now, where it works, where it completely breaks, and what developers are getting wrong.

• We talk about why front-end and UX matter more than ever in an AI world, why most people misunderstand agents, and what real day-to-day workflows with LLMs actually look like.

• There’s also a hard look at who benefits from AI, who falls behind, and whether we are quietly building fragile systems that we don’t fully understand.

If you’re a developer trying to figure out where this is all going, this is one of those conversations worth paying attention to.

Expect to learn:

• Why AI is as much a UX problem as it is a backend problem

• What “agents” actually mean in practice, not in demos

• Where LLM workflows are useful today and where they fail hard

• Whether junior developers should be worried or excited

• How building apps changes when AI is part of the system

• What developers should actually be doing right now to stay relevant

Neil also has a podcast, The Skill Tree, on AI and agentic-specific topics.

We also get into a bigger question most people are avoiding:

• Are we heading toward AI-assisted coding… or AI-orchestrated systems where developers become supervisors?

• And maybe more importantly… which side of that shift do you want to be on?

For years, I've wondered at the intricate designs of MAKE files, venvs, poetry, binaries, uvs, yml, and many other such spells that engineers through the ages have deemed necessary to provide a consistent runtime environment. Project to project, company to company, it all changes.

It’s a mix of this and that; the goal is clear, the tools are not.

You and I want an easy-to-use, continuous, and production-like environment to weave our spells and write our code. Well, we used to cast our code; Claudious does that now. It’s a question of reduced friction at the end of the day. We want to handle our software and data in a way that replicates as closely as possible the systems the code will crawl.

The times are changing, like it or not. The price tag to produce a line of code has come down a lot; it’s been cheapened. Yeah, I hear ya. There will always be people and places who are willing to pay for, and desire, the hand-wrought in a forge of blood and sweat … code.

So if the LLMs are spewing out mid-level drivel at an ever-increasing pace, what’s left for the old and well-worn data persons, like me, us, you?

Listen, you scalleywag, yeah, you! You think I’m over here recording podcasts, writing stuff, taken a beat’n for you out of the goodness of my heart??! Ha!

Get 50% Off for 1 Year!

Get your moldy wallet out, you heartless pirate. %50 off for a year, throw a guy a bone.

After code … infrastructure and containers.

It seems everyone is mourning the death of the programmer; maybe it’s too early to place the headstone and chisel out the date … I feel like a poor soul on the Batavia, harassed by storms and thrown ashore on Abrolhos.

Except I’m a writer of code, some 397 years later, the ship is my life, the storm is AI, and the island is the world at large.

I have a feeling it feels worse when you’re a helpless player in a cosmic theme than it is in reality. But then again, I’m currently gainfully employed and have not yet had to face that daunting task of throwing myself upon the shoals of LinkedIn’s job board.

AI has shifted the ground underneath our feet; methinks you can’t put the cat back in the box, even if you want to, and I’m not saying I do. Best to just trudge ahead and figure out the new world, feeling our way around in the dark, hoping someone or something doesn’t hack off your fingers.

I have noticed some things, though.

Things that give me hope for the future, and maybe a path to walk.

The end is not here … yet.

Sure, every look into the crystal ball of the future to divine what comes next usually comes to naught. But I am willing to step into the breach for you and make some predictions. It’s a free world; I’m free to be wrong and confident in my assertions pulled from my own experience.

I have the benefit of already being a coder for decades, and now have the opportunity to use AI and LLMs to expand what I can do and the speed at which I do it, without necessarily losing or not learning a bunch of skills. Although I could see that happening over longer periods of time.

At the day job, I build a lot of products and features from scratch, literally, from a C-suit or Product produced One-Pager with an idea, to a delivered piece of software in production. This gives me unique insight into how AI is affecting the “build” process as a whole.

I’m just going to jump straight into it, take what you will from it. The following are takeaways I have observed and experienced firsthand regarding the New Age of AI Software Development: building products from conception to reality.

• The entire production timeline has shrunk significantly.

• “The Business” is more technical now (they use AI).

• Architecture and systems design have now become the bottleneck.

• Decision-making by people, humans, is also a new bottleneck.

• Writing “the code” has become the easy part.

• More time is spent on the infrastructure.

• Quick, seamless CI/CD deployment cycles are becoming increasingly important.

• Containerization is important.

The game has changed when it comes to developing and producing software products, and I don’t think in a bad way. In fact, it gives me hope for the future of software. We still need experienced programmers who are good at their job. But their job can no longer be just spewing code.

Claude can spew code, we don’t need code spewers, we need humans who are very technical, and can bridge the gap between some vibe-coded non-reality produced by a Product Manager, and a well-oiled system that can run a thing in production, long term.

We could sum up the findings I listed above like this …

The code is arguably the easy part, call it what you want, easier to produce, cheaper to write, more or less … code production that WAS the bottleneck 5 years ago, is now the flip-flop to be the fastest and quickest part of a software project.

• Does this mean we no longer need programmers?

No, in fact, I find it amplifies the value of the senior engineer who has built a set of skills over a long period of time, some technical and some non-technical, skills that aren’t easy to develop.

Let’s break down these three high level findings about building end-to-end products in the age of AI.

Share

The Human Bottleneck

What a strange way to think about the new bottlenecks of AI-assisted development workflows. It’s hard to forecast how it will all shake out in the future, but clearly, based on my experience with the full lifecycle of this Brave New World, humans indeed still play a critical part.

Yeah, I know there is a broad spectrum of “AI usage” in the real world, ranging from never-touched-it to Gas Town. The vast majority of AI usage will fall squarely in the middle, of course. The noisiest folk online will be in either extreme camps, the silent majority in the middle.

My comments on this subject apply to the middle group. I can’t help people who bury their heads in the sand and think that there is no need to ever touch AI. I also don’t have much to say to folk who Gas Town every project, they will reap what they sow.

That being said, I’ve noticed that humans are indeed the new bottleneck, albiet a critically important part of the process, we are always the slow part.

• Humans are still, in great part, the “idea factories.”

• Humans are always procrastinators and indecisive

• Most humans lack focus

• Communication is always a problem and hard

Think about it this way: for those who have started using AI to assist with the generation of the “technical part” of Software projects, what was once the bottleneck has now become waiting for other humans to handle the above bullet points.

It wasn’t such a big deal in the past, when the C-suit and Product Managers were meeting, talking, ideating, researching, etc, because Engineering was happily over in the corner doing MVPs, POCs, and generally moving at a snail’s pace.

Now it has reversed.

The truth is, Engineering can, if they want, pump out MVPs, POCs, and near-production-ready systems like hotcakes. That’s not the long part anymore. If they are good, they spend a few days on architecture and systems design, and shovel out the technical implementation at a rapid pace, and iteration is even faster than before.

The bottlenecks are indeed more human-centric.

• We have to wait for the C-suit to come up with the next “big thing.”

• We have to wait for Product (or whoever fills that role) and the C-suit to battle it out.

• We have to wait for them to do their customer and market research, and validation.

• Engineering leaders have to go back and forth with said people to hammer out the possibilities.

• Engineering has to build and iterate, feeding back info into this human loop, waiting for updates and changes, ready to spew technology.

It has simply become less of a process of waiting weeks or months for engineers to hammer out those bits and bytes. That has been shortened considerably. The long part is the waiting for humans to make decisions, research, ideate, communicate, etc.

Systems design, architecture, and infrastructure.

The second piece of the software lifecycle puzzle that has become more important, something that doesn’t take more time than before, but because of the pace of code production, simply appears to be a larger part of the process, is systems design, architecture, and infrastructure. All rolled into one.

Truth be told, long before AI came along to write all your functions, the planning and design part of the software project was always the most critical.

The only thing that has changed is that now it has 10x’d in importance.

Why?

Because those legions of Claude Engineers are highly susceptible to thinking less and doing more (code), on a whim and fancy, because they simply CAN move faster, and maybe even feel pressured to move faster.

It’s the new norm. In a feint to appear at the top of their game, on board and forward thinking … they will just start to prompt their way into a solution before anyone has taken the human step of simply … thinking.

• It’s clear some engineers think that putting Claude in “Plan Mode” is all they have to do.

I mean, you can wing it if you want, just trust the LLM that’s able to predict the next best token … with the future of your Data Platform or Software project, but Lord be with you at the end of that road.

What does this all mean in the end?

You should be the person spending time sitting and considering the …

• Systems Design

• Architecture

• Infastructure

Look, I don’t care if you talk to an LLM about the different options available to you. Pros and cons, what’s hot and what's not. How does this fit into that? In fact, that’s a good idea, it helps you not miss the sharp and hidden edges.

But, you should be filtering these decisions through your humaness. You should think about the Engineers around you, what the goals and desires of your CTO are, what other projects and priorties are coming down the pike in the neaer future of Product Management.

To simply make technology decisions, or let the LLM make these technology decisions purely based on … some probability of the next right token … is insane.

• What are the skills and traits of your engineers?

• What are the budget and time constraints?

• How does this fit into next year’s goals?

• What meets the needs of the business?

Systems design, architecture, and infrastructure are the domain of smart engineers who care about their craft and have seen firsthand decades' worth of right and wrong decisions borne out in reality.

I personally feel like AI simply makes someone 10x what they already were.

Careless design and thoughtless foresight have been the bane of many a system since time began, nothing has changed, the problem has only accelrated with AI.

I brought up a few technical points and shifts I have experienced firsthand while working in a software lifecycle that includes AI. I want to bring this story I’ve written you home by emphasizing a few practical points and takeaways.

Practical Applications

Again, you have listened to me rant about enough if you have made it this far, for which I commend you, but you might be asking, “So what? Where does that leave us, me?”

Good question.

Here are some practical technical takeaways for folks living inside an AI-driven software development lifecycle. Things I have found have changed or become very important.

If you have been left wondering recently what your future might look like, whether you will have a job, and how you can differentiate yourself since code is no longer the blocker? If I were you, I would use this list as a study guide.

• More time is spent on the infrastructure and systems design.

• Quick, seamless CI/CD deployment cycles are becoming increasingly important.

• Containerization is key.

• DevOps is hot again.

• Terraform … aka YML as code is non-negotiable.

This might look like a disjoined list of things, but I think they are more related than you think, owing to the fact that they stem from the fundamental shifts AI has wrought on a lot of software development lifecycles, not all, but a lot.

When code is produced in a fraction of the time, what was once an ancillary task becomes an important part. If someone produces an MVP in a matter of hours or days, then the movement of that software product into some infrastructure, on some computer, through some automated pipeline is now the difference between make or break.

• Systems Design and Infrastructure are now the all-important stage on which the play is danced.

• Deployment pipelines are the conduit through which the glut of new features and products is pushed into the world.

• Containers are easy to use, quick to encapsulate, and deploy a variety of software and tools we now swim in.

• Infra as Code (IaC) is the bedrock upon which the castles of sand are built.

Where once you found your value in the intricate lines of hand-crafted and artisanal functions with which you dazzled your peers, eventually LLMs will rip this from your withered fingers, in part, not in whole, but in part.

Set yourself apart by going the extra mile, Systems Design and Architecture, DevOps and CI/CD, IaC, and Containers. An acute awareness that humans still have a role to play in this pipeline is a valuable skill to refine.

I urge you to be different in this age of speed, shipping code at lightning speed; you should slow down. Consider the business and human context in which you find yourself designing software solutions.

Andreas shares his full journey from industrial IoT and working at Bosch to building one of the largest data engineering education platforms in the world, training over 2,000 students and reaching more than 100,000 engineers globally. We get into what production data systems actually look like, why most learning paths are broken, and how AI is reshaping the role of the modern data engineer.

• We also dig into the uncomfortable truths. AI can write code, but it cannot replace thinking. Most engineers focus too much on tools and not enough on problem-solving, system design, and communication. That gap is only getting bigger.

If you are trying to figure out how to stay relevant in data engineering, or you are just getting started and want to avoid years of wasted effort, this conversation will change how you think about your career.

Today’s podcast is sponsored by Estuary.

Without them, content like this isn’t possible. The best way to support this Newsletter is to check out what Estuary has to offer and click the links below.

Build millisecond-latency, scalable, future-proof data pipelines in minutes.

Estuary is the Right-Time Data Platform that integrates all of the systems you use to produce, process, and consume data. Also, providing best-in-class CDC (Change Data Capture).

Estuary unifies today’s batch and streaming paradigms so that your systems, current and future, are synchronized around the same datasets, updating in milliseconds.

What we cover:

• Why most data engineers are learning the wrong things

• The shift from coding to problem-solving and system design

• How AI is actually changing data engineering workflows

• Why courses and tutorials are becoming less effective

• The difference between real production systems and “toy projects.”

• The future of data engineering jobs and whether AI will replace them

• Why fundamentals still matter more than ever

One of the biggest takeaways is simple. The tools will keep changing, but the problems stay the same. The engineers who win are those who understand systems, ask better questions, and connect business problems to real solutions.

Links:

• Learn Data Engineering Academy:

https://learndataengineering.com

• Andreas Kretz on LinkedIn

• Andreas Kretz on YouTube

• Sponsor: https://estuary.dev

A luddite’s observations on AI

Somehow, I still conduct my day job without relying on AI regularly. As your friendly neighborhood Anonymous Rust Dev, you might think I’ve found a home for it at this point, but thus far, I’m just as much of a cantankerous old git as the actual creator of Git.

I’ve had people wax poetic, tell me these incredible success stories, and yet they haven’t won me over.

So, what’s holding me back? Am I just resisting the inexorable tides of change? What’s actually going on in my head? As I buck this trend and the peer pressure that goes along with it, am I just setting myself up for failure?

The observations of a Millennial

As a self-described Luddite, I think it is important to remember the original Luddites:

The Luddites were members of a 19th-century movement of English textile workers who opposed the use of certain types of automated machinery due to concerns relating to worker pay and output quality.
-(Wikipedia)

The Industrial Revolution may be the single greatest acceleration of societal change across history. We have made numerous advancements as a species, many of them game-changers, but access to industrial processes irrevocably changed how we function. The Luddites had good reason to fear the new paradigm — it was a fundamental attack on their way of life. And yet, they were on the leading edge of a societal evolution that would eventually come to introduce a new “middle” class and an era of prosperity for the Western world.

...But, at what cost?

The modern era is built on untold horrors arising from industrialism. Today, the idea of unions rubs so many the wrong way, but they are a product of a massive power imbalance that existed between workers and employers.

The likes of Carnegie, insanely wealthy and famous for his philanthropy, reached that point atop the backs of mistreated laborers. It is often joked that OSHA regulations are written in blood, and those who bristle at having to follow seemingly stupid rules should remember that, in their absence, employers could have their workers doing some really nasty and dangerous work, with little to no recourse.

Honestly, I can’t begin to do this topic justice. I don’t think we’re ignorant of these horrors; anyone who’s played games in the Bioshock or Fallout franchises, for instance, can see the [fictional] course trajectories of worlds where the guard rails weren’t there.

Old School

Born in the early 80’s myself, I’ve watched the world change at a rapid pace. My early memories were of things like the family’s Ford LTD, VCR decks, the Nintendo Entertainment System, and eventually I’d even get to a classroom environment where the class had its own dedicated Apple IIe in the corner of the room.

I’d read the works of futurists who said that one day every house would have a computer, and not long after, news broadcasts would do specials about how the Information Superhighway was coming to life and would soon touch our lives.

I don’t doubt that the generation before me was already clinging on for dear life. My grandmother could scarcely turn on the TV with the remote, and we’d be over at her house teaching her which one of the goofy symbols on the remote would start or stop her movie.

Society is still reeling, I think. We haven’t found footing, and old people today struggle to discern fact from fiction when scrolling through their Facebook feeds. My generation, along for the described ride during our formative years, has enough perspective and adaptability to barely keep up with it, if only just.

And the newest generations are being thrust into a world of adult-oriented haptics like touchscreens and voice recognition, or neatly digested information in YouTube Shorts that lets them bypass the learning process that earlier generations were forced to contend with, and I can’t help but wonder what it’s doing to hobble their developmental progress.

Paradigm shifts have been a thing for the last couple of hundred years, but their frequency is increasing at an alarming rate. We’re being tossed about in ever choppier seas, and at some point, I expect the whole thing to capsize.

And then, we see the appearance of generative AI. The pace at which this technology evolves and its impact on the general populace are at once exciting and frightening.

People who don’t truly understand the technology get to witness this magic genie that seemingly has an answer for everything, while the people who do understand it (insofar as any one of us truly can) know that it’s simply stringing together the works of yore. This tech lands at the worst possible time, when we’re still reeling from the collective shock of the pandemic and dealing with tumultuous sociopolitical upheaval and a deepening divide across multiple facets of society.

The observations of a freelancer

For much of this, I watch from the sidelines. I have no business overlord forcing AI down my throat; I can only watch with fascination as many businesses, like Google or Facebook, attempt to mandate that their workers use AI in their day-to-day activities. For them, the tool is a shiny new hammer, and they’re going to find a nail eventually.

It doesn’t seem to be going all that well, though...

Meanwhile, I stay in touch with old colleagues in the web development business. They’ve got it rough — one of the areas in which generative AI excels is in displacing creativity, and digital design studios are especially hard-hit.

Of course, we all know it’s soulless crap, but it’s becoming increasingly difficult to discern the difference between the real and the fiction.

At this point, I honestly don’t know how the real Will Smith feels about spaghetti, and if he came out in a video showing and telling us his love for the stuff, I wouldn’t be able to tell you whether it was true. r/isthisAI gives a glimpse of the struggle to see through the haze.

...And personally, I find it very hard to tell anymore what’s fake and what’s authentic. I like to think I know what I’m looking for, but the cues are increasingly hard to detect. At least in my own generation, I feel we’ve been sufficiently primed to expect digital misinformation indistinguishable from fact; how much worse for the older generation, with their difficulties keeping up, or for the younger ones who haven’t seen the evolutionary path and are simply being dropped right into the midst of this?

I think it’s all by design. We’re currently so inundated as a society with change that we can’t hope to keep up. It’s a functional Gish gallop — we’re assaulted from all sides, and we can only devote so much energy to making sense of it all. Hidden in this assault, I think, are multiple concerted efforts:

First, businesses clearly see an opportunity to displace workers or readjust the power dynamic in their own favor. Unlike the fantasy universe of Wall-E, though, we probably can’t expect the automation to work in favor of the proletariat... Instead, the chasm between the haves and the have-nots will only widen.

• For a brief period during the pandemic, the IT worker was a highly coveted asset that businesses would fight over; now, big business sees a rare opportunity to flip the script and devalue those same workers by claiming AI can do their jobs. Whether that’s true or not, the narrative is being cemented, and soon we should count ourselves “lucky” if a business deigns to give us a job and the time of day to go with it.

Second, we’re being conditioned by AI vendors. Free-tier and low-cost AI services were a gateway drug, giving people a taste of what’s possible. As workers increasingly depend on AI to do the dirty work for them and consequently lose the skills or muscle memory associated with that work, there will one day be a rug-pull in which that subscription cost balloons rapidly, and people who got hooked will need to keep their drug dealers happy no matter the price.

• The vibe coding era will be quickly replaced by a “prompt specialist” one, where you can no longer just pick up the tech on a lark, but instead need to sell your soul for access to it.

Third, in a rather conspiratorial vein, I also see an emerging technocracy, where certain leaders in the tech space will have positioned themselves to profit from the chaos. I don’t need to name them; you probably already know who I mean, but they will have the keys to the kingdom.

• They will own the data centers, dictate policy, and be the final arbiters of right and wrong. It’s already happening, and will only increase in its efforts over time. There is still a shrinking window of opportunity to fight this, as the digital arms race collides with economic and infrastructure difficulties, but if they emerge victorious, then we’ll be facing a grim future.

The observations of a developer

So, do I think it’s all horrible? No, quite the contrary — I see a place for tooling to emerge that can function as a force multiplier, for those who know what they’re looking at and how to accommodate it. Granted, anyone who depends on AI-generated code needs to question the ethics involved; those models aren’t above plagiarizing the works of those who came before.

The major dangers I see are:

• As already stated, there’s a hook of reliance and an atrophy of skills for people who depend too heavily on this stuff

• The illusion that developers are easily replaced can create economic shock

  • The idea that developers are easily replaced by automation will displace entry-level talent, and when they’re not given the opportunity for growth, the future talent pool will be effectively choked out

  • Talented engineers will be needed to come in and fix the mess when the dust settles, but their work will be much harder as they’re forced to comb through an accumulation of slop and codebases absent of a coherent vision

Read more