Quasi-Agentic Pipelines with Databricks and Apache Airflow

the strange space in between

Yet another day passes with me casting a lazy eye over the endless dribble of marketing-produced content on the once-glorious but now fallen LinkedIn; at the intersection of Agentic/LLMs and Data, if you allow me to call it that. The winds of change are blowing hard, and it’s difficult to decipher the real from the classic hello-world demo tricks. A wise man once said that “all things are possible, not all things are profitable.”

What we really need, what I need, is more real-life examples of building quasi-agentic pipelines (data) that walk the thin red line down the middle of the road. Most of us have no desire for the Gas Town and Loop your Prompt Bros that hammer at us incessantly at every waking hour. There is also a very real need for Data Engineers and data practitioners generally to accept the world has changed, and we need to provide useful solutions to the business; some of those include providing the ability to add LLMs/Agents into our existing data workflows.

Well, so add a Task to the pipeline that calls an API endpoint that just happens to be an LLM. Isn’t it that easy? Yes and No … of course.

I wanted to show and talk about two things today that may or may not be interesting to you. Mostly as a way of providing a third point of view into the morass of Agentic discourse, once that is grounded more in reality, while slowly trying to incorporate new things into old tools. A practical approach to agents/LLMs inside data workflows and pipelines.

1. Using the same old tools … Databricks and Apache Airflow to solve new Agentic problems that the business wants solutions to.

2. Talking about determinism in data pipelines and bridging the gap between what the business wants you to do and how you do it as an Engineer and Architect.

One will be moderately technical, the other philosophical and *gasp political.

---

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

---

The Agentic “problem” in a world that hasn’t caught up.

We are going to start backward today. When introducing today's plan and discussion, I listed the whole “… determinism in data pipelines and bridging the gap between what the business wants…” as part deux, but I’m going to start from the bottom up and talk about this first because it sets the stage for the more technical solution that I’ve used to solve this very real business request. First, it isn’t uncommon in the Brave New World that we live in to get some version of this request from “The Business” or Product … or the C-Suite; it doesn’t really matter. We do what we are told.

I mean, as I said, it’s a new world we live in where our most gracious and wonderful business counterparts have learned via ChatGPT and Claude that the sky is the limit. If they can dream it up, apparently we can build it. The issue with this dream of randomly injecting some AI into the middle of an otherwise normal workflow is not new, but is fraught with difficulty and has bred an entire army of consultants and new “AI Tool(s)” to do this for us.

So what are you waiting for? Done yet?

With everything that has changed … one thing has not. The last %10 of that “easy” project is where the blood is spilled onto the ground.

• We can’t buy and implement a new AI tool for every new use case.

• We HAVE to work within the constraints of our existing infrastructure and systems.

• We need to deliver reasonable results that meet expectations on a reasonable timeline.

The answer to every question involving AI isn’t simply another tool, package, subscription, some new backflip, and throwing salt under the old oak tree at midnight. We can’t redesign the entire stack to send rockets to the moon because someone simply wants an LLM/Agentic feature to be added at some random spot. This is not the way. We need to find the right balance of being problem solvers inside an engineering and business context. This means we are thoughtful, considerate of cost and time, and focused on good-enough solutions that are a great starting point and don’t involve signing up for some new year-long SaaS contract just to call an endpoint that has an LLM behind it.

---

Apache Airflow + Databricks for LLM integration.

Ok, so I just wanted to cover this topic, which won’t be that exciting, but does contain some interesting talking points that we will get to. Recently, at the day job, I’ve been working on integrating LLMs into your average, everyday data pipelines in a reasonable manner, without trying to boil the ocean or start over. As an engineer, when I start to solve a problem, I first turn to the tools that I have in front of me. I’m reluctant to add new tools and infra to an existing stack without good reason.

Core tenets are …

• simplicity

• dealing with non-determinism

• use existing tooling until it doesn’t work

This a real data flow inside a Production Airflow DAG. It’s a mix of normal, Python-centric data tasks and API calls to an LLM that may or may not return what you expect, knowing that downstream tasks depend on these non-deterministic tasks. Sorta like taking your life in your own hands, ya know. “Claude, don’t screw this up, no mistakes.”

Anywho, not sure if you keep up to date on the happenings at Databricks; it can be hard to keep pace. One of the most unsung heroes and cool features on Databricks is modeling serving endpoints. Databricks does things well, like it or not, and they make it super simple to create and serve API endpoints for LLMs that you can call with a simple TOKEN.

For example, serving an LLM behind an API sounds super complicated, right? This is a sample from a production Terraform file/repo that runs the LLM behind the API calls above, which go to a Databricks Modeling Serving Endpoint. All batteries and bolts included. It’s amazing that in a few short years we went from the cool kids playing with LLMs to a few lines of code that will spin up a model-serving endpoint ready for use. Simplicity is key; architecturally, implementing features that aren't a burden to the overall system allows us to move on and start providing value for our business.

I won’t bore you with the Python DAG and Python function details; they are far from special. Calling a Databricks Model Serving Endpoint from an Airflow DAG is like calling any other API … the only fizzbuzz being that you aren’t ever really going to be sure what comes out the other end; you have to design for that ambiguity and non-determinism.

Again, the point is that you don’t need some fancy new tool(s) to add “Agentic” capabilities to your data pipelines. Sure, if you think you need to rebuild self-healing pipelines run and managed by agents, then yeah, you probably do need to go buy some SaaS. But if you have Databricks and Airflow, two of the most well-known tools known to human-data-kind, then yes, with a little fiddling around, you can add LLM calls into your data pipelines. The biggest problem posed with doing this, as I mentioned a few times above, is how to deal with the non-determinism.

Most of us, alot of us, deal in Directed Acyclic Graphs. We have units of work, tasks; they have relationships with other tasks, dependencies, etc. This is, in theory, in direct opposition to inserting LLMs into the mix. What, at the core, is a DAG?

• Directed: Lines have arrows that show a one-way path / strict dependency from one node to another node.

• Acyclic: There are no loops … aka feedback cycles.

• Graph: A collection of tasks connected together.

We have to build some sort of verification into the system; at least we should, unless we are going to close our eyes and let ‘er rip, come what may. Which isn’t the worst idea in the world. I mean who knows what we are asking the LLM to do, it could be any number of things. Clean data, check data, make data, look for this, find that, combine this. Whatever. But, unlike the rest of our DAG, which is most likely a very strict set of code that will return a very specific thing … we can’t really say for certain an LLM will do what we ask. At least not 100%

Indeed, a brave new world.

We aren’t the only ones thinking about this.

To be sure, even if you don’t hear talking heads talking about this one, we aren’t the only ones mulling over what it means to combine Agents/LLMs with classic data tooling. Apache Airflow itself has deemed it necessary to release a Common AI Provider.

They provide the ability to make single Task Decorator LLM calls.

from pydantic import BaseModel
from airflow.providers.common.compat.sdk import dag, task

@dag
def my_pipeline():
    class Entities(BaseModel):
        names: list[str]
        locations: list[str]

    @task.llm(
        llm_conn_id="my_openai_conn",
        system_prompt="Extract named entities.",
        output_type=Entities,
    )
    def extract(text: str):
        return f"Extract entities from: {text}"

    extract("Alice visited Paris and met Bob in London.")

my_pipeline()

Even multi-task Agent steps.

from airflow.providers.common.ai.toolsets.sql import SQLToolset
from airflow.providers.common.compat.sdk import dag, task

@dag
def sql_analyst():
    @task.agent(
        llm_conn_id="my_openai_conn",
        system_prompt="You are a SQL analyst. Use tools to answer questions with data.",
        toolsets=[
            SQLToolset(
                db_conn_id="postgres_default",
                allowed_tables=["customers", "orders"],
                max_rows=20,
            )
        ],
    )
    def analyze(question: str):
        return f"Answer this question about our data: {question}"

    analyze("What are the top 5 customers by order count?")

sql_analyst()

And possibly the scariest of all: LLM Branching, which lets the Agent decide which downstream tasks to run. Yikes!!

@task.llm_branch(
    llm_conn_id="my_openai_conn",
    system_prompt="Classify the support ticket priority.",
)
def route_ticket(ticket_text: str):
    return f"Classify this ticket: {ticket_text}"

Heck, they even added Human-in-the-Loop gates!

LLMOperator(
    task_id="summarize_report",
    prompt="Summarize the quarterly financial report for stakeholders.",
    llm_conn_id="my_openai_conn",
    require_approval=True,
    approval_timeout=timedelta(hours=24),
    allow_modifications=True,  # reviewer can edit the output
)

Wow, things get a little complicated fast, I methinks. Who knows what the future holds; time will tell. Even though the world of AI seems old, everyone is Clauding their way to glory, as far as REAL Agentic work that gets to be integrated into our existing tech stacks; well, I would argue that we are still in the infant stage of that. Lots of new tooling is being released into the data world for agents; only time will tell what sticks and what doesn’t.

Also, as more time goes by and demand for LLM-integrated data pipelines grows, the community will speak at large about what it likes, doesn’t like, and the best ways to move forward with integrating tools like DAGs and LLMs. Without experience running these sorts of pipelines in production, it’s hard to say what the sharp edges and pitfalls are that are hidden under the water.

I personally think that big question isn’t …

“How do we insert LLM/Agents into pipelines.”

But rather …

“How do normal determanstic pipelines integrate with LLM/Agents that are non-determanstic.”

What I mean by this is that it’s one thing to have a purely LLM/Agent pipeline running, because all the tasks are deemed non-deterministic and encapsulated together. It’s something else to mix the two. Like I did. Kept a guy up a night, ya know. It’s walking the fine line between trusting an inherently untrustworthy entity and what were once trustworthy, deterministic pipelines. No more. This is even more important in a Data World. Trust in data is the cornerstone of many a business these days, and the downfall of many. The unknowns of LLMs/Agents getting their hands on the data pipelines definitely raise some interesting questions.

Have you been playing with LLM/Agentic tasks inside your existing data pipelines? How has it gone? What tools are you using? What do you think the future holds?