An Empirical Evaluation of Columnar Storage Formats

Xinyu Zeng, Yulong Hui, Jiahong Shen, Andrew Pavlo, Wes McKinney and Huanchen Zhang (Tsinghua / CMU / Voltron Data), 2023 (VLDB). T1, and the benchmark half of this spoke’s format coverage edge. Note the authors: Pavlo runs CMU’s database group, McKinney created Arrow and pandas — this is not a disinterested comparison of two formats so much as an argument about what the next one should look like.

What they did

Stress-tested parquet and orc — “the two most widely used open-source columnar storage formats” — across varied workload configurations, measuring both performance and space.

What they found

The framing is the finding: both formats were “developed over a decade ago, in the early 2010s, for the Hadoop ecosystem,” and the hardware and workloads have moved. Their design recommendations are each a criticism of a current default:

• Use dictionary encoding by default, rather than as an option to enable.
• Favour decoding speed over compression ratio for integer encodings — the old trade assumed disk was the bottleneck and CPU was not.
• Make block compression optional, because it costs more than it saves in some modern setups.
• Embed finer-grained auxiliary structures so queries can skip more precisely.

They also report inefficiencies “when handling common machine learning workloads and using GPUs for decoding” — two consumers that did not exist when these formats were designed and that now read a large share of the world’s columnar data.

What it means for this spoke

The lakehouse story (storage-compute-disaggregation) rests on an open format at rest that any engine can read. This paper says the open formats everyone standardized on encode assumptions from the Hadoop era — spinning disks, CPU-bound decompression, no GPUs, no training loops — and that newer engines are therefore leaving performance on the table by conforming to them.

That is a genuine tension with the disaggregation argument rather than a footnote: the portability that makes shared storage possible is also what freezes a decade-old design in place. An engine can improve its execution freely and cannot improve the format without leaving the ecosystem. Flagged in synthesis.