Multi-Head Latent Attention

At the end of the last article I said there was a fourth answer to the KV-cache problem.

Here it is.

Multi-Head Latent Attention, introduced in DeepSeek-V2, does not shrink the cache by sharing key and value heads the way MQA and GQA do. It shrinks the cache by compressing the information those heads would have stored into a smaller latent vector, and caching that instead.

Same goal. Different mechanism. And the mechanism interacts awkwardly with RoPE, which is why it deserves its own article.

The problem we already know

During autoregressive generation we cache keys and values so we do not recompute them for every new token. That cache grows with sequence length, layers, heads, and concurrent requests.

When the cache is large, two things go wrong at once:

• GPU HBM fills up, so long contexts and large batches stop fitting.
• Decode becomes memory-bandwidth bound. Generating the next token means reading the whole cache out of HBM. If the cache is fat, the GPU spends its time waiting for memory, not multiplying.

MHA stores a full K and a full V for every head and every token. That is the expensive default.

MLA's question is simple: what if we do not store the expanded forms at all?

Compress first, expand later

In standard attention:

$$K = XW^{K}, \qquad V = XW^{V}$$

In MLA, the model first projects the hidden state into a lower-dimensional latent:

$$c^{KV} = XW^{DKV}$$

where

$$c^{KV} \in \mathbb{R}^{d_c}, \qquad d_c \ll d_k + d_v$$

$W^{DKV}$ is a learned down-projection. $c^{KV}$ is the compressed KV latent. That latent is what goes into the cache.

When attention needs keys and values, learned up-projections rebuild them:

$$K^{C} = c^{KV}W^{UK}, \qquad V = c^{KV}W^{UV}$$

The superscript $C$ marks the content component of the key. We will need that distinction in a moment.

The important distinction:

The cache stores $c^{KV}$, not the full $K^{C}$ and $V$.

A useful analogy: imagine every token wants a 1,000-dimensional key/value story, but the story that matters fits in 256 dimensions. Cache the 256. Expand back to the larger spaces only when attention needs them.

This is not a generic compressor bolted onto a finished model. The down-projection, the up-projections, and the attention path are trained together. The model learns representations that survive the bottleneck.

Why RoPE makes this harder

Transformers need positional information. Otherwise these two sentences collapse into each other:

The cat chased the dog.
The dog chased the cat.

Modern models usually inject that information with RoPE: rotate query and key vectors by an angle that depends on position.

If you compress the key and then apply RoPE to the compressed form, the order of operations fights you. RoPE is position-dependent. A linear transform of a rotated vector is not generally the same as rotating the compressed vector. If you want attention to be computable directly from the cached latent, you need the positional path to stay compatible with that math.

MLA's answer is to decouple the two jobs.

Split the key:

$$K = [K^{C};\; K^{R}]$$

and the query the same way:

$$Q = [Q^{C};\; Q^{R}]$$

Then the attention score factors:

$$QK^{T} = Q^{C}(K^{C})^{T} + Q^{R}(K^{R})^{T}$$

Content attention can live on the compressed latent path. Positional attention lives on a separate RoPE track that is not forced through the same bottleneck in a way that breaks the rotation.

That split is not an implementation detail. It is the architectural choice that makes low-rank KV compression coexist with rotary embeddings.

A concrete size comparison

Take a simplified layer:

• 16 query heads
• key dim per head: 128
• value dim per head: 128
• compressed latent: 256
• RoPE key dim per head: 32

Standard MHA caches:

$$16 \times 128 + 16 \times 128 = 4096$$

numbers per token.

A simplified MLA-style cache stores:

$$256 + 16 \times 32 = 768$$

numbers per token.

That is about an 81% reduction in this toy setup. Real models use different widths, and the exact saving depends on those widths. The point of the example is the shape of the trade: store a shared latent plus a thin positional component, not full multi-head K and V.

As context length and batch size grow, that difference stops being academic.

Why this matters on a GPU

Decode still has to compute attention. MLA does not remove the softmax or the weighted sum of values.

What it changes is how much state you keep around between tokens.

A smaller cache means:

• more concurrent sequences fit in HBM
• less traffic from HBM into the compute path on every decode step
• long-context serving becomes less of a memory tax

In other words: same attention equation, cheaper residual state.

MLA is not GQA with a new name

It helps to put the four variants next to each other.

	Main idea
MHA	Each query head has its own K/V heads
MQA	All query heads share one K/V head
GQA	Groups of query heads share K/V heads
MLA	Compress KV into a learned latent, keep position separate

MQA and GQA shrink the cache by sharing.

MLA shrinks the cache by compressing.

Both are inference-economy moves. They are not the same move.

What "low-rank" is doing here

Suppose a full key lives in 1,024 dimensions. Not every dimension is independent information. A learned projection can map the useful part into a smaller space:

$$c^{KV} = XW^{DKV}$$

That is the same spirit as low-rank factorization:

$$W \approx W_{1}W_{2}$$

Instead of storing or applying one large transform, you route through a narrower intermediate. In MLA, that intermediate is what the KV cache holds, and the attention path is designed around it.

Does compression hurt quality?

It can.

If $d_c$ is too small, information needed for attention is lost. If you take a finished MHA model and squash its cache after the fact, you should expect pain.

MLA is different because the model is trained with the bottleneck from the start. The representations adapt to the latent width. The real question is not "does compression lose information?" It does, in principle. The engineering question is:

Can we compress enough to save memory while keeping the attention behavior the model needs?

Latent dimension, training recipe, and how RoPE is decoupled all sit on that tradeoff.

The deeper pattern

Standard attention treats the KV cache as a warehouse of the exact keys and values future tokens will use.

MLA treats the cache as a warehouse of a compact sufficient statistic for those keys and values.

That pattern shows up everywhere in systems and machine learning: do not store the expanded form if a smaller representation preserves what the next operation needs. The hard part is making the compression compatible with the next operation.

For MLA, compatibility means three things at once:

1. Content attention still works from the latent.
2. Positional behavior survives via the decoupled RoPE path.
3. Decode stays cheap because the cached object is small.

Takeaway

Multi-Head Latent Attention makes Transformer inference more memory-efficient by caching a learned low-rank KV latent instead of full multi-head keys and values, and by keeping RoPE on a separate track so the score math still factors cleanly.

The flow to remember:

hidden state X
      ↓
KV down-projection
      ↓
compressed latent c^KV   ← this is cached
      ↓
learned up-projections
      ↓
content keys K^C and values V
      ↓
attention  (+ separate RoPE score term)

It does not remove attention. It changes what attention is allowed to remember between tokens.