The creator of Pandas and co-creator of Apache Arrow, Wes McKinney, joins the Data Engineering Central Podcast for an in-depth conversation about how modern data engineering came to exist, where AI is taking software development, and why good engineering still matters more than ever.
We start with Wes’ journey from building GoldenEye fan websites as a teenager to creating Pandas while working at a quantitative hedge fund, and eventually launching Apache Arrow, one of the foundational technologies behind today’s modern data ecosystem. Along the way, we discuss Cloudera, Parquet, DuckDB, DataFusion, Spark, and how the industry evolved from Hadoop to today’s lakehouse architectures.
The second half of the conversation dives deep into AI. Wes explains why large language models make experienced engineers more productive but won’t magically replace software engineering, why architecture and good taste are becoming more valuable than writing individual lines of code, and why projects like DuckDB and
Apache Arrow remains incredibly difficult to recreate with AI alone. We also discuss open-source, local AI models, token costs, multimodal data platforms, and what new engineers should focus on to build long-term careers in software and data.
If you’re a data engineer, software engineer, architect, engineering leader, or simply interested in where AI is taking our industry, this is a conversation you won’t want to miss.
Topics We Cover
How Pandas was created
The story behind Apache Arrow
Why Arrow became the standard for modern data systems
DuckDB, DataFusion, and the next generation of data tools
The evolution from Hadoop to lakehouses
Why AI won’t replace great software engineers
Architecture vs. coding in the AI era
Building trustworthy open source software
The future of data engineering
Advice for new engineers entering the industry
If you enjoy conversations with the people building the future of data engineering, subscribe for more interviews with the creators of the tools we use every day.
