Multi-Head Latent Attention (MLA)

DeepSeek's answer to the kv-cache memory problem, introduced with DeepSeek-V2. Instead of caching a key and a value vector per head per token, cache one shared low-rank latent vector per token and reconstruct what each head needs from it. Derived in full in serret-attention-primer; the DeepSeek-V2 report itself is not held here.

The construction

Write the per-head projections as one wide matrix each — W_Q, W_K, W_V are the column-wise concatenations of the heads' matrices. MLA then factors them through low-rank latent subspaces of dimension d_L (for keys and values, shared) and d_LQ (for queries):

W_K = W_L · W_LK        W_V = W_L · W_LV        W_Q = W_LQ · W_LQQ

with W_L shared between keys and values. The cached object becomes L = X·W_L, one vector of dimension d_L per token, for every head at once. In DeepSeek V2 that is d_L = 512 against 128 heads — the reason the model has no "KV heads" row in serret-attention-primer's comparison table at all. The concept does not apply.

An implementation note the paper is careful about: DeepSeek trains the latent model directly rather than deriving the factorization from a trained MHA model. The algebra above explains what the architecture is, not how it was obtained.

Two merges are where the compute win comes from

The factorization would be pointless if every step had to expand the latent back into per-head K and V. It does not, because two products can be pre-multiplied into single matrices:

Query side. In Q_h·K_hᵀ the latent-to-query weights and the latent-to-key weights meet, so W_LQQ_h · W_LK_hᵀ collapses into one matrix W_LQK_h. Scores are computed straight from the latents.
Output side. The latent-to-value weights merge with the head-combining output projection W_O into a single W_LO.

After both merges only W_L, W_LQ, W_LQK and W_LO are needed, and only the latent vectors are held in memory. Since d_L and d_LQ are smaller than d_in, the merged form is cheaper to compute as well as cheaper to store — the rare case where the memory optimization is not paid for in FLOPs.

The RoPE problem, which is the part usually skipped

Without positional embeddings, MLA is exactly equivalent to a GQA/MHA model. The factorizations are algebraic identities, so nothing is approximated and nothing is lost.

Rotary position embeddings break that. RoPE is applied after K is built, which inserts a position-dependent matrix R_m between W_LQQ_h and W_LK_h — exactly between the two factors that merge 1 above depends on. Keeping RoPE in its usual place would therefore not merely forfeit the speedup: it would require reapplying the positional embedding at every evaluation, adding overhead.

DeepSeek's fix is to split the vectors — append a non-latent portion to the key and query, and apply RoPE only there. The merge survives on the latent part, so the computational advantage is kept and the exact equivalence to MHA is deliberately given up. This is the cleanest instance in the spoke of an efficiency mechanism whose real constraint turns out to be its interaction with an unrelated architectural choice, rather than anything about the mechanism itself.

TransMLA (Meng, Yao & Zhang, arXiv:2502.07864, "Multi-Head Latent Attention Is All You Need") is cited as a method for approximately converting already-trained GQA/MHA models into the latent form — which would make DeepSeek's optimization applicable to weights DeepSeek did not train. Not held here; worth a source.

Against the alternatives

grouped-query-attention reduces how many K/V vectors are cached by sharing heads. quantization reduces the bytes per element. MLA changes what the cached object is — a single shared latent instead of per-head keys and values — and is the only one of the three that also lowers compute, via the merges. It is also by far the most invasive: GQA and cache quantization can be applied to an existing design, while MLA is trained in from the start unless something like TransMLA converts it.

What is not established here

No numbers. This corpus holds no measurement of MLA's cache saving, throughput, or quality against a GQA baseline — the storage comparison in serret-attention-primer is counted in floats, not timed on hardware, and the source reports no experiment. The DeepSeek-V2 paper is the missing primary and is named on the growth edge in synthesis.

attention-mechanism · kv-cache · grouped-query-attention · serret-attention-primer · quantization · llm-inference · synthesis