Loving Polars and Delta Lake - but not sure I was say "goodbye" to Spark as there are still a lot of scenarios that benefit massively from it. But then again, I am a tad biased, eh?! :)
Great post as usual Daniel, I also have a small project with same stack with your blogpost where I ingest a parquet from S3, transform it to delta table, do some processing and export back to S3.
And Polars just got Iceberg support: https://github.com/pola-rs/polars/pull/10375
Further supporting your argument that the future has a heavier focus on developer experience.
Loving Polars and Delta Lake - but not sure I was say "goodbye" to Spark as there are still a lot of scenarios that benefit massively from it. But then again, I am a tad biased, eh?! :)
You know I was just trolling you with this entire post right? :)
It certainly did cross my mind :)
I am on your boat. I do feel good about Polars still i feel there's no reason to say goodbye to Spark
Great post as usual Daniel, I also have a small project with same stack with your blogpost where I ingest a parquet from S3, transform it to delta table, do some processing and export back to S3.
Very cool!
Love Polars. It's even better if you mix it with DuckDB.