Apache Parquet

The on-disk columnar format most of this spoke’s engines read and write. From the project’s own format documentation (T1, first-party).

Layout. A file opens with the 4-byte magic PAR1, then column chunks grouped into row groups — “N columns in this table, split into M row groups” — and closes with a footer holding “the locations of all the column chunk start locations”, a 4-byte length, and PAR1 again. Metadata last is a deliberate choice: it permits single-pass writing, because you only know the chunk offsets once the data is written. It also means a reader’s first move is a seek to the end.

Separation of metadata from data is explicit in the design, so columns can be split across files or a central metadata file can reference many Parquet files. Supported alongside: compression, multiple encodings, nested data, bloom filters, encryption and checksums.

Why the spoke needs it

clickhouse, duckdb and every lakehouse claim in storage-compute-disaggregation assume a format where a query can read three columns out of two hundred and skip the rest. Parquet is what makes that concrete: column chunks give projection, footer statistics give row-group skipping, and the format’s independence from any engine is what allows storage and compute to be sold separately at all. The table formats in coverage edge 5 (Iceberg, Delta, Hudi) are metadata layers over files like these — they do not replace them.

And it is not modern. columnar-format-evaluation (Zeng et al.) stress-tests Parquet against ORC and finds both were “developed over a decade ago, in the early 2010s, for the Hadoop ecosystem,” with design choices that no longer match current hardware — the detail is on that page. Read this spec as the incumbent standard, not as the best available design.

Contrast apache-arrow, which is the in-memory counterpart: Parquet optimizes bytes at rest, Arrow optimizes access in RAM.