I hate to say it because it makes me feel old, but I’ve been writing PySpark code for … hmmm … I don't know, a long time. I mean, I was writing blogs about Spark back in 2018, it’s fair to say I was playing with it well before then.
Say, maybe 7-10 years or something? That’s long enough, and I still don’t consider myself a Spark expert by any stretch of the imagination, even if I have been writing it full-time for years and at the 100TB+ level. That should tell you something about yours truly.
Keep reading with a 7-day free trial
Subscribe to Data Engineering Central to keep reading this post and get 7 days of free access to the full post archives.