Apache Parquet

The on-disk columnar format most of this spoke's engines read and write. From the project's own format documentation (T1, first-party).

Layout. A file opens with the 4-byte magic PAR1, then column chunks grouped into row groups — "N columns in this table, split into M row groups" — and closes with a footer holding "the locations of all the column chunk start locations", a 4-byte length, and PAR1 again. Metadata last is a deliberate choice: it permits single-pass writing, because you only know the chunk offsets once the data is written. It also means a reader's first move is a seek to the end.

Separation of metadata from data is explicit in the design, so columns can be split across files or a central metadata file can reference many Parquet files. Supported alongside: compression, multiple encodings, nested data, bloom filters, encryption and checksums.

Why the spoke needs it

clickhouse, duckdb and every lakehouse claim in storage-compute-disaggregation assume a format where a query can read three columns out of two hundred and skip the rest. Parquet is what makes that concrete: column chunks give projection, footer statistics give row-group skipping, and the format's independence from any engine is what allows storage and compute to be sold separately at all. The table formats in coverage edge 5 (Iceberg, Delta, Hudi) are metadata layers over files like these — they do not replace them.

And it is not modern. columnar-format-evaluation (Zeng et al.) stress-tests Parquet against ORC and finds both were "developed over a decade ago, in the early 2010s, for the Hadoop ecosystem," with design choices that no longer match current hardware — the detail is on that page. Read this spec as the incumbent standard, not as the best available design.

Contrast apache-arrow, which is the in-memory counterpart: Parquet optimizes bytes at rest, Arrow optimizes access in RAM.

The cloud-era critique (added 2026-08-22)

btrblocks (SIGMOD 2023) attacks Parquet on a metric this page did not have. Measured the usual way — against uncompressed size — every Parquet variant decompresses at over 50 GB/s, comfortably above a 100 Gbit link. Measured against compressed size, which is what actually arrives over the wire, Parquet+Zstd manages 24.8 Gbit/s and the scan goes CPU-bound. Two structural causes are named: a fixed encoding set chosen by hard-coded rules, and the metadata footer, which forces three dependent S3 requests to read one column. The paper's own verdict on fixing it is that a "Parquet v3" carrying BtrBlocks' encodings "would not share much with the original besides the name."