Protobuf wire format

How protocol-buffers turns a message into bytes. The whole design follows from one decision: the encoding stores field numbers, never field names or types, so the bytes are meaningless without the .proto that produced them. Everything the format buys — the size, the speed, the evolution rules — is paid for with that dependency.

Tag, then value

Every field is a varint tag followed by a payload. The tag packs two things:

tag = (field_number << 3) | wire_type

so the low three bits are the wire type and the rest is the field number. Three bits allows eight wire types and six exist:

LEN is the one that makes the format recursive: a varint length follows the tag, then that many bytes. An embedded message is just a LEN payload, which is why nesting costs nothing structural.

Varints, base 128

Integers are variable-width, one to ten bytes. Each byte gives seven payload bits plus a continuation bit in the MSB; the payloads are appended together, least-significant group first. The docs' worked example is 150, encoded as 96 01:

10010110 00000001   raw bytes
 0010110  0000001   drop the continuation bits
 0000001  0010110   reverse the groups
   → 10010110 = 128 + 16 + 4 + 2 = 150

Small numbers are cheap and large ones are not, which is why field numbers 1–15 fit in a one-byte tag and the docs push you to spend them on your hottest fields.

The negative-number trap

int32 and int64 encode negatives as two's complement widened to 64 bits, so -2 costs all ten bytes. sint32/sint64 avoid that with ZigZag, which interleaves the sign into the low bit:

sint32: (n << 1) ^ (n >> 31)
sint64: (n << 1) ^ (n >> 63)

Positive p becomes 2p, negative n becomes 2|n| - 1. Same declared range, five times the bytes if you pick the wrong one, and nothing in the type name warns you — a real footgun sitting in plain sight in a T1 doc.

Repeated fields

Since Edition 2023, repeated fields of primitive type are packed by default: one LEN record holding the concatenated values rather than a tag per element. Under protobuf-editions this is the repeated_field_encoding feature and can be set back to EXPANDED per field, which is exactly the kind of behaviour editions exist to make adjustable.

Why this belongs next to the spoke's other wire formats

clickhouse-rowbinary measured RowBinary at 2.1–3.3× faster than JSON and found developers abandoning it anyway, because the format is unforgiving about types and fails silently on 64-bit values. Protobuf's answer to that class of problem is the generated code: you never hand-write the parser, so the type mismatch is caught at compile time in your language rather than at 3am in production. yaff goes the other way and deletes the parse step entirely, keeping .proto as the schema while replacing the runtime representation with an mmap-able flat layout — a rejection of exactly the tag-and-varint scan described on this page.

Related

protocol-buffers · protobuf-editions · grpc · clickhouse-rowbinary · yaff · developer-tooling · synthesis