Retrofitting language models to operate over bytes

Minixhofer, Murray, Limisiewicz, Korhonen, Zettlemoyer, Smith, Ponti, Soldaini & Hofmann (Allen Institute for AI, Cambridge, UW, Imperial College London, LMU Munich), Nature, published online 2026-10-05 (DOI 10.1038/s41586-026-11111-4). Received via Telegram PDF. The first source here on how a tokenizer could be avoided entirely rather than on how one is built — a different answer to the coverage gap this spoke named, not the one it asked for.

The method: byteification, not training from scratch

Every prior byte-level LLM in this space was trained from random initialization and compared against a subword LLM also trained from scratch — a race the byte-level side has consistently lost, because frontier subword training keeps absorbing unrelated advances (data curation, architecture, post-training) the byte-level work can't track on a research budget. This paper instead retrofits an existing subword checkpoint into a byte-level model, using under 1% of a typical pretraining budget (49.1B tokens total). The resulting models: Bolmo 7B / 1B (from Olmo 3 7B / OLMo 2 1B), Bwen 8B (from Qwen3 8B Base), Blama 8B (from Llama 3 8B).

The architecture: a latent tokenizer language model (LTLM)

A local encoder operates directly on bytes; a non-causal neural boundary predictor (it can see future bytes, unlike prior work) decides where to place patch boundaries; a pooling step reduces each patch to one representation, contextualized by a deep global transformer, then depooled back to bytes and decoded by a local decoder that predicts the next byte. The non-causal boundary predictor is the paper's stated departure from earlier latent-tokenizer architectures (such as the Byte Latent Transformer): seeing future context lets the model place boundaries that are both accurate and faithful to how the source subword tokenizer would have segmented the same text, where prior causal-only approaches had to trade one property off against the other.

Training runs in two stages: stage 1 (~M² FLOPs) is a cheap warm start that gets a checkpoint close to the source model's performance fast, shortening the experimentation loop; stage 2 (~3x FLOPs of the full model) is the full backpropagation pass. Stage 1 helped the 1B model more than the 7B, suggesting (not confirmed) that larger models tolerate skipping the cheap warm-start stage better.

Results against the two things that matter

Against subword baselines, byteified models are close, not better or worse by much. Bwen 8B scores 69.8 on the paper's 7B byteification-suite average against Qwen3 8B's own 70.3 (difference not statistically significant, p≈0.69); Blama 8B and Bolmo 7B show similarly small, mostly non-significant gaps to their own sources.

Against prior byte-level models trained from scratch, byteified models win by a wide margin. EvaByte 6.5B, TFree-HAT 7B and BLT 7B (all random-init byte-level baselines) trail on the same suite average by double digits (45.7 / 56.5 / 48.9 vs 59.8-70.3 for the byteified models); Bolmo 7B posts a +16.5 absolute point improvement over BLT 7B on STEM tasks specifically.

On character-level tasks, byteification doesn't just match the source — it beats it. On the paper's Char/CUTE/EXECUTE suite, Bwen 8B averages 78.9 against Qwen3 8B's own 66.2; Blama 8B scores 81.0 against Llama 3 8B's 57.2. The byte representation is doing real work here, not just avoiding a regression — this is the one place the method clearly outperforms its own source model, consistent with the standing argument that subword tokenization specifically costs character-level understanding.

Code is mixed. Pass@1 sometimes drops slightly versus the subword source (Bwen 39.6 vs Qwen3's own 45.8); pass@16 usually recovers and exceeds it (Bwen 69.0 vs 60.7) — more samples closes more of the gap than one does, which the paper doesn't explain further.

Inference speed: claimed, not quantified in the main text

The conclusion states the byteified models "come close to matching subword-level LLMs in inference speed" and that byte-level models can be sped up further by training with a higher bytes-per-patch ratio, something not available to the same degree in subword models. No wall-clock or tokens-per- second figure for this claim survived PDF text extraction — it is likely stated only in a figure (Extended Data Fig. 3, mLSTM throughput) rather than in prose. Recorded as a claim pending a number, not as a measured result on this page.

What this does and doesn't settle for the spoke

This doesn't answer coverage edge 8 as written — "how tokenizers are built" (BPE, SentencePiece, vocabulary construction) is still unsourced here. What it supplies instead is the competing answer to the same underlying question: a tokenizer step can be skipped altogether, at a retrofitting cost under 1% of pretraining, while staying close to subword-level performance and clearly ahead on character-level tasks. tokenization's own finding — that tokenization is a fixed ~19ms cost and a model-family compatibility boundary — would both become moot inside an LTLM, since there is no separate tokenizer step and no vocabulary to mismatch across checkpoints.

Standing

T1 — peer-reviewed Nature, multi-institution (AI2 primary), code and models stated as released alongside the paper. Single paper on this exact method; no independent replication yet, and the one quantitative inference-speed claim this spoke most wants is not extractable from the main text.

tokenization · llm-inference · token-sampling · kv-cache · allen-institute-for-ai