Why does KV cache grow with context and concurrency?

The bottleneck

Autoregressive decoding repeatedly reads keys and values for past tokens. Caching avoids repeated projections, but every request and layer retains history. Longer inputs and outputs and higher concurrency increase memory and bandwidth demand.

The mechanism

For standard MHA/MQA/GQA, ideal KV payload is 2×B×L×T×H_kv×d×s bytes. The 2 counts K and V; B is equal-length requests, L layers, T cached tokens per request, H_kv KV heads, d head dimension and s bytes per element. Sum individual lengths for unequal requests.

KV_MiB = 2 B L T H_kv d s / 2²⁰

Examples and evidence

Default teaching configuration B=1, L=32, T=4096, H_kv=8, d=128, s=2 gives 512 MiB; eight such requests need 4096 MiB. The calculator excludes weights, activations, workspace, fragmentation, quantization metadata and replicas.

New constraints

Smaller capacity does not imply proportional speedup: decode may be limited by bandwidth, compute or scheduling. MLA has a different compressed representation; this standard formula is not its exact memory model.