AI is NEVER going to take your job.

AI ain't going to get me.

I’m going to give you the spoiler up front. AI isn’t taking your job anytime soon. If you think otherwise, you’ve been watching too much YouTube and reading the wrong stuff on LinkedIn. Heck, maybe you simply HAVEN’T been using AI enough on more than junior-level problems.

• AI is good at writing boring, easy code.

• AI is terrible at new or cutting-edge things.

• AI itself requires massive amounts of Data Engineering.

• AI lets you focus on other things (arguably the more important things)

The truth is I wish AI were better, those talking heads keep saying how every benchmark gets better and better, but the reality on the ground is much different.

Can AI do a lot of non-value-added things in a programming context? Of course, it can, just like it can do a lot of mundane tasks in many areas.

• AI, make me a marketing plan for X, Y, Z.

• AI, write me an email or context that says this or that.

• AI, write me this Python script.

The internet, particularly GitHub, is full of horrible ideas and code. AI has learned this along with everything else, being not the wiser.

Yes, AI (Cursor, CoPilot, ChatGPT, whatever) can write code like there is no tomorrow.

It’s not good code.

It can’t solve complicated problems that spill across many different areas. It isn’t very good at remotely new topics (like the last few years).

Yes, AI can write you Postgres queries, SQL queries, and a Spark data pipelines.

I did a little test.

I was recently working on a project that is all very obvious, that is, how poor AI is at non-trivial tasks. I will summarize it as much as possible.

• Using Spark code to combine various datasets

  • CSV, TXT, Delta Lake, etc

• Get pgvector setup on PostgreSQL

• Get code to combine various datasets and create embeddings

• Store the embeddings into the pgvector table

• Query data back out in RAG application

• Write a RAG application using recent tooling

Basically, AI was slightly worse than a junior developer at doing ALL these things. Just to be nice, I spent about a day on it, tweaking prompts and giving it links to blogs where pieces of the solution were in play.

It was just too much.

Sure, if I had asked AI how to make me a doubly linked list in Python, it would have spit it out with no problem. Real-life stuff? Not going to happen in a million years.

AI totally sucked at the following things.

• understanding how to use newish tools like Polars, DuckDB, Daft for processing data.

• understanding new storage tools like Delta Lake and how to interact with them.

• how to use a newish technology like pgvector

  • probably a combo of too new and too obtuse

• How to write performant Python that could interact with Postgres in a way that …

  • didn’t OOM

  • would finish in this century

• Understanding the recent developments in building RAGs

  • things like langchain

  • interacting with embedding and chat endpoints

And the list goes on.

AI will trick you.

The funny thing is AI will trick itself and you into thinking it actually knows what it’s doing. It will be confidently wrong and close at the same time.

Prompt with some errors?

“Sure, let me fix that for you, I know what’s wrong.”

I’m confident that tweaking prompts and inputing errors for a week would have not gotten you to the end, and the result would have been a blob of code you could never fix or understand.

Because you didn’t write it.