Apache Arrow

The columnar format for data in memory, not on disk — the distinction this page exists to make. From the specification itself (T1): “The Arrow columnar format includes a language-agnostic in-memory data structure specification, metadata serialization, and a protocol for serialization and generic data transport.”

Layout

Every array is a validity bitmap plus one or more data buffers plus metadata (length, null count, type). The bitmap encodes nullability one bit per slot — “a 1 (set bit) for index j indicates that the value is not null.” Fixed-width types get a single values buffer; variable-length types get offsets plus values; nested types hold child arrays. Buffers are aligned so that access is SIMD-friendly and random access is constant time by pointer arithmetic.

The property that makes it infrastructure rather than a library detail: arrays are “relocatable without ‘pointer swizzling’, allowing for true zero-copy access in shared memory.” Two processes, or two languages, can read the same buffer with no serialization step. The IPC message format (continuation marker, Flatbuffers metadata, 8-byte-aligned body) is how record batches move between them.

The trade it names

The spec is explicit that this is not free: Arrow “provides analytical performance and data locality guarantees in exchange for comparatively more expensive mutation operations.” A columnar in-memory layout is for reading and scanning, not for updating rows.

Against Parquet and ORC

parquet and orc compress bytes for storage; Arrow lays bytes out for the CPU. That is why a system reads Parquet from object storage and materializes Arrow in memory rather than choosing between them — and why the format boundary is where duckdb and clickhouse hand data to Python, R or a Rust process without paying a conversion. storage-compute-disaggregation needs both halves: a portable format at rest so any engine can read the data, and a portable format in RAM so the engine’s output is not locked to it either.