Tokenization

Tokenization turns the input bytes into the integer IDs that llm-inference actually runs on. It happens before prefill, and everything the rest of this wiki measures — prompt cost, kv-cache size, tokens per second — is denominated in its output.

Thin page, and deliberately so. The spoke has no source on how tokenizers work (BPE, SentencePiece, vocabulary construction); that is still coverage edge 8. What is sourced here is the step's behaviour in a running pipeline, from one measurement okf-token-handoff. byteification (2026-10-07) answers a different question the edge didn't ask — not how a tokenizer is built, but how to have none at all — by retrofitting an existing subword model to operate directly on bytes. Both the fixed ~19ms cost below and the vocabulary-compatibility boundary are specific to the tokenizer-as-separate-step design; an LTLM has neither.

A second source restates the rule from the outside: openmetal-tokens-per-second notes English averages ~1.3–2 tokens per word, that punctuation and whitespace usually cost their own tokens, and that code tokenizes differently again — so the denominator of every tokens-per-second figure is model-specific and workload-specific at once.

A fixed ~19 ms, whatever the model

Running a modern Rust-backed BPE tokenizer over a few thousand tokens costs on the order of 19 ms, measured as the difference in time-to-first-token between doing it and skipping it on two Qwen2.5-Coder checkpoints okf-token-handoff. The same ~19 ms showed up for both models, so it reads as a fixed cost of the input rather than something that scales with the model. Expressed as a percentage it looks bigger on the smaller model (37.8% vs 28.0%) purely because the forward pass it is being compared against is shorter.

Against an end-to-end run it is small — under a tenth of a percent of a 41.3-second three-agent pipeline. It shows up where first-token latency is the thing being watched.

The vocabulary is a compatibility boundary

Token IDs mean nothing outside the vocabulary that minted them. Two models can share IDs only if their vocabularies are identical entry for entry, which is why okf-token-handoff's pipeline compares the full ~151,936-entry vocabulary dictionaries before one agent accepts another's integers. A mismatch does not crash: the receiving model reads the wrong subwords and produces fluent, confident nonsense.

This makes the tokenizer a model-family boundary in any system that passes tokens rather than text between models. Same family, verified per checkpoint: shareable. Across families: not.

Skipping it is a reuse mechanism

Pre-tokenizing once and handing the array downstream belongs with the wiki's other don't-redo-work levers rather than with its make-work-cheaper ones. It sits one step upstream of prefix caching (coverage edge 7), which reuses the kv-cache computed from a shared prompt; this reuses the integers computed from shared bytes.

Related

llm-inference · token-sampling · kv-cache · okf-token-handoff · byteification