This is a most misrepresented article on two fronts
1. tested column pruning and the dataset you access would have been 2 columns + metadata for the parquet files so probably fit in memory even without streaming.
2. Most of the processing time would be IO bound on S3 and the access patterns/simultaneous connection limits etc. would have more of an impact than any processing code.
Love that you went through the pain of trying the different systems but I'd like to see an actual larger than memory query.
Interesting read, but the conclusion about Spark feels a bit misleading. Benchmarking a distributed engine in single-node mode and then calling it “slow” ignores the entire reason Spark exists in the first place. Engines like DuckDB, Polars, and Daft are purpose-built for single-node performance; Spark is not.
Your results are valid for highlighting how far single-node systems have come, but they don’t really say much about Spark’s value proposition. It’s like testing a race car in a supermarket parking lot and being surprised it doesn’t shine...
You could have repartitioned the data as you went to take advantage of Spark better... with as many cores across as many machines as you want. You don't quite make this clear.
This is a most misrepresented article on two fronts
1. tested column pruning and the dataset you access would have been 2 columns + metadata for the parquet files so probably fit in memory even without streaming.
2. Most of the processing time would be IO bound on S3 and the access patterns/simultaneous connection limits etc. would have more of an impact than any processing code.
Love that you went through the pain of trying the different systems but I'd like to see an actual larger than memory query.
Go read someone else's Substack, you hobbit. There's a whole wide world out there that doesn't include me.
Hi, you could test combining duckdb and polars as both uses arrow.
I/O using duckdb (support delete vectors) and then transform using polars. Perhaps a good combo?
I've been meaning to write more on using pyarrow for data interchange and reading/writing https://www.confessionsofadataguy.com/pyarrow-for-large-dataset-processing/ I will add it to my list for this substack
Great article, very interesting results
Thanks, appreciate that
Interesting read, but the conclusion about Spark feels a bit misleading. Benchmarking a distributed engine in single-node mode and then calling it “slow” ignores the entire reason Spark exists in the first place. Engines like DuckDB, Polars, and Daft are purpose-built for single-node performance; Spark is not.
Your results are valid for highlighting how far single-node systems have come, but they don’t really say much about Spark’s value proposition. It’s like testing a race car in a supermarket parking lot and being surprised it doesn’t shine...
You could have repartitioned the data as you went to take advantage of Spark better... with as many cores across as many machines as you want. You don't quite make this clear.
Curious how Presto/Trino/Athena stacks up
Also would be interested to see total requests/data transferred/CPU usage with each tool as well
DuckDb has a new "DuckLake" catalog format that would be another candidate to try. https://ducklake.select/
Great article, love the series so far!
At 0.61 per hour, your request cost you 36USD :')
That's why people should become paid subscribers!!! Hahahaha
Which version of DuckDB?
just UV add duckdb, so whatever that pulled
Damn, it would have been nice to have chdb here.
Good article, thanks