Spark vs Polars. Real-life Test Case.

[Chris Guecia]

Great write up!

I too hit that wall of sink_parquet not being able to write partitions.

Regardless, polars is very fun to use. Coming from spark and pandas I really like how familiar it feels.

[Edgar]

Hi Daniel,

Might be worth to rerun this following this guide:

https://luminousmen.com/post/how-to-speed-up-spark-jobs-on-small-test-datasets

BR

E