Apache DataFusion is Now the Fastest Single Node Engine for Querying Apache Parquet Files

Apache DataFusion 43.0.0 has become the fastest single node engine for querying Apache Parquet files in ClickBench, surpassing DuckDB and chDB/Clickhouse using the same hardware. This marks the first time a Rust-based engine holds the top spot, which was previously held by traditional C/C++ based engines. DataFusion's open design allows users to start quickly with a full-featured Query Engine and customize any behavior needed. The performance improvements were achieved through various techniques such as using Arrow StringView, optimizing Parquet file reading, skipping partial aggregation when it doesn't help, and optimized multi-column grouping.