Attention mechanism

The operation that makes a Transformer a Transformer, and the reason the KV cache exists. The spoke covered its consequences — the cache, its IO cost, its memory management — for two months before holding the mechanism itself; serret-attention-primer supplies it.

A database lookup with a soft match

The cleanest statement of what attention does: a database is a set of (key, value) pairs, and a query returns the value whose key it matches. Attention is that with the equality relaxed into a similarity. Given queries and a set of key/value pairs, the output for each query is a weighted sum of all the value vectors, weighted by how well each key matches that query.

Three linear maps produce the operands from the same embedded tokens — Q = X_Q·W_Q, K = X_K·W_K, V = X_V·W_V — so "query", "key" and "value" are three learned views of the same text, not three different inputs. The output dimension is the value dimension.

Softmax is a choice of kernel, not the definition

serret-attention-primer writes the similarity as an arbitrary kernel κ(v,w), usually of the form f(⟨v,w⟩), computes the score matrix A = κ(QKᵀ), normalizes each row to sum to 1, and multiplies by V. Standard softmax attention is the case

κ(v,w) = exp(⟨v,w⟩ / √d_QK)

and the normalization then is the softmax. The √d_QK is not decoration: if the entries of v and w are i.i.d. with mean 0 and variance 1, then ⟨v,w⟩ has variance d_QK, so dividing by √d_QK returns the score to unit variance before exponentiating.

Reading it this way separates the structure (normalize, weight, sum) from the similarity function, and that separation is what makes the variants legible. grouped-query-attention and latent-attention change which matrices exist and which get stored; the operation above is untouched in both. It is also the framing under linear-attention and randomized-kernel work — the paper's own context is an IPAM workshop on randomized numerical linear algebra — though this spoke holds no source on that yet.

Multiple heads

Several attention operations run in parallel on the same input with their own W_Q, W_K, W_V, and their outputs are concatenated and passed through an output projection W_O. The stated purpose is to let different heads attend to different kinds of relation. Mechanically the heads are independent until the concatenation, which is what allows the sharing in grouped-query-attention and is why the KV cache is sized per head.

Masking is how causality is imposed

A decoder must not let a token attend to tokens after it. That is done with a mask added to QKᵀ before the kernel — −∞ for softmax attention, so those positions get exactly zero probability after exponentiation — or equivalently by zeroing the masked entries of the score matrix afterwards. This is a constraint on the attention pattern, not a separate mechanism, and it is the reason autoregressive generation can reuse everything it has already computed.

Self, cross, streaming, full

Self-attention — queries, keys and values all come from the same sequence. This is what decoder-only models such as GPT use throughout.
Cross-attention — queries come from one sequence, keys and values from another. The original Transformer's decoder uses it to see the encoder's output; serret-attention-primer notes it is also the root of multimodal models, where text and image embeddings meet in a shared space.
Streaming attention — the query tokens are a subset of the key/value tokens: only the new tokens need queries, because everything earlier is already represented in the cached K and V. This is the regime an inference server actually runs in.
Full attention — every token queries every token. What prefill does, and what streaming avoids repeating.

Why it costs what it costs

Attention is O(N_Q · N_KV · (d_QK + d_V)), and in generation d_QK and d_V are small next to the token counts — so the token counts are the whole story. Two facts follow, and the spoke's central constraint sits between them: the last query token's output depends on the K and V of every prior token, and the query vectors of prior tokens are never needed again. So keys and values are worth keeping and queries are not, which is exactly kv-cache, and the memory it takes is what grouped-query-attention and latent-attention were built to reduce.

serret-attention-primer · kv-cache · grouped-query-attention · latent-attention · flash-attention · llm-inference · tokenization · synthesis