Understanding Transformers and Attention Mechanisms: An Introduction for Applied Mathematicians

Michel Fabrice Serret (Center for Scientific Computing, Theory and Data, Paul Scherrer Institute), arXiv:2604.00965v1 [math.NA], 1 April 2026, 13 pages. Written as an introductory presentation for the IPAM Randomized Numerical Linear Algebra workshop, project "Randomization in Transformer models."

Read locally from the PDF with pypdf. It runs from tokenization and embedding through attention, multi-head attention, masking, the encoder/decoder Transformer and its BERT/GPT descendants, and finishes on the part this spoke cares about most: KV caching, Grouped Query Attention and Latent Attention as answers to the memory cost of the cache.

T2, not T1. It is a preprint, unreviewed, and it reports no new result and no measurement — an expository derivation written for an audience of applied mathematicians. What makes it worth its grade is that the derivations are complete and the author is not selling anything. For the primaries this spoke still wants, see the growth edge in synthesis: Ainslie et al. for GQA and the DeepSeek-V2 report for MLA.

The framing that makes it useful here

Attention is set up as a database lookup with a soft match: a set of (key, value) pairs, a query, and an output that is the value vectors combined in proportion to how well each key matches. Formally the similarity is an arbitrary kernel κ, and softmax attention is one instance of it — κ(v,w) = exp(⟨v,w⟩/√d_QK), with the √d_QK there because ⟨v,w⟩ has variance d_QK when the entries are i.i.d. mean-zero unit-variance, so the scaling puts the score back at variance 1.

Separating the structure (normalize, weight, sum) from the choice of kernel is the paper's pedagogical move, and it is the one that pays off later: latent-attention and grouped-query-attention then read as changes to which matrices exist and get cached, with the attention operation itself untouched. Held on attention-mechanism.

What it says about the KV cache, with the arithmetic

The paper states the cost this spoke's kv-cache page describes qualitatively:

• Recomputing key and value vectors at every new token costs O(N³·d) over a conversation.
• Caching them makes each new token linear, so O(N_Q·N_tokens·d) per new sentence and O(N²·d) overall.
• The price is memory: 2·N_L·N_h·N_KV·d floats, times the bytes per float — layers × heads × cached tokens × dimension, twice over for K and V. "This can quickly become prohibitive, especially in the long-context case."

It also gives a name this spoke lacked: streaming attention is attention where the query tokens are a subset of the key/value tokens, as against full attention — the regime that makes the cache necessary in the first place.

Three real model configurations

Table 3 puts numbers on the abstraction, which is the kind of grounding the mechanism pages here usually lack:

Llama 3 and Gemma 3 are GQA models — note Llama 3 sharing 64 query heads over 8 KV heads while Gemma 3 uses 16 over 32, a 8:1 against a 2:1 ratio for the same d_head. DeepSeek V2 has no KV-head row because latent attention removes the concept: one shared latent vector of dimension 512 per token, for all 128 heads.

The sharpest technical point: MLA and RoPE do not compose

Worth quoting the finding, because it is usually skipped in secondary accounts. In the absence of positional embeddings, a latent-attention model can be written exactly as an equivalent GQA/MHA model, because the low-rank factorizations are algebraic identities. Apply RoPE and the equivalence is lost — RoPE inserts a position-dependent matrix R_m between the latent-to-query and latent-to-key weights, which is precisely the product latent-attention needs to pre-merge. Merging is what buys the speedup, so keeping RoPE in the ordinary place would not just fail to speed things up, it would add overhead by requiring the positional embedding to be reapplied every evaluation.

DeepSeek's answer is to append a non-latent part to the key and query vectors and put the positional embedding only there. The compute advantage survives; the exact equivalence does not. The paper also points at TransMLA (Meng, Yao & Zhang, arXiv:2502.07864) for approximately converting already-trained GQA/MHA models into the latent form, which would let DeepSeek's optimization be applied to other people's weights.

What it does not do

No experiment, no benchmark, no wall-clock or memory measurement of its own. The complexity and storage results are counted, not timed, so nothing here speaks to what any of it costs on hardware — which is the axis cloudflare-kimi-glm-serving and flash-attention-paper supply and this does not. It also stops at the architecture: nothing on serving, batching, scheduling or quantization.

Related

attention-mechanism · grouped-query-attention · latent-attention · kv-cache · llm-inference · flash-attention · tokenization · synthesis