How do MQA, GQA and MLA reduce KV?
Shared KV heads and latent compression are distinct designs.
Content revision 4 · 2026-10-24
I02 · Revision 1 · 2026-10-21
- MHA: KV per query head
- GQA / MQA: shared KV
- MLA: latent + positional components
The bottleneck
Query heads can attend differently, but independent K/V for every query head increases cache. MQA (2019) and GQA (2023) explore shared K/V; DeepSeek’s MLA changes the cached representation.
The mechanism
MHA has H_kv=H_q. MQA shares one KV set across query heads, H_kv=1. GQA uses an intermediate group count. MLA uses a low-dimensional joint latent representation, reconstruction mappings and positional components. It is not simply a smaller GQA group count.
MHA → GQA: ideal KV ratio = H_kv / H_q
Examples and evidence
With 32 query heads and 8 KV heads, GQA has one quarter of MHA’s ideal payload, all else equal. Meta discloses GQA in Llama 3 8B/70B, and DeepSeek-V3 discloses MLA. These are model-specific examples.
New constraints
Sharing trades against model quality and requires training or conversion and validation. Latent representations need architecture and kernel support. A serving setting cannot losslessly turn any model into MLA, and payload ratios do not determine throughput ratios.
Primary sources
- Fast Transformer Decoding: One Write-Head is All You Need · 2019-11-05 · Checked 2026-10-21
- GQA: Training Generalized Multi-Query Transformer Models · 2023-05-22 · Checked 2026-10-21
- Meta Llama 3 · 2024-04-18 · Checked 2026-10-21
- DeepSeek-V3 Technical Report · 2024-12-27 · Checked 2026-10-21
Manual browser reading records and interactive calculators are available with JavaScript.
Keep exploring
I01 · Why does KV cache grow with context and concurrency?