Induction head

A two-layer attention circuit that completes a repeated pattern: having seen … A, B … A, it predicts B. It is the mechanism behind in-context learning of token statistics, and prior work reports it appearing abruptly during training and carrying a causal role in in-context learning across model scales. Documented here from phase-transitions-in-attention.

The copy sub-circuit is the hard half

The first layer performs the copy or shift-by-one operation, [… A, B] → A: attend to the previous position and carry it forward. Earlier proofs identify learning that first layer as the main difficulty, which is why phase-transitions-in-attention isolates it — a single-layer network trained on a supervised copy task, with the induction head’s second layer left out entirely.

The theory’s three stages match the empirically reported learning progression in two-layer transformers: attention contributes nothing, then all context is attended equally (matching in-context unigram statistics), then a single position is attended (matching bigram statistics).

Why this spoke holds it

The head is an attention mechanism, so the vocabulary overlaps llm-inference-wiki, which owns attention as a serving concern (kernels, KV cache). Nothing about induction heads here is about serving: the subject is how the circuit forms during training, which is this spoke’s side of the line. Cross-link, don’t duplicate.

Related

phase-transitions-in-attention · capability-emergence · machine-learning · bayesian-inference