When should prefill and decode be disaggregated?
Compare independent scaling with state-transfer cost.
Content revision 4 · 2026-10-24
S06 · Revision 1 · 2026-10-21
- Encode: image / video encoding
- Prefill: inputs → KV
- Transfer KV state
- Decode: stepwise generation
The bottleneck
Prefill processes inputs in parallel; decode generates stepwise. They have different compute/bandwidth needs and latency targets, and may interfere when colocated. DistServe studied separation in 2024; NVIDIA Dynamo disclosed PD in 2025 and discussed multimodal EPD conditions in 2026.
The mechanism
PD assigns input processing and generation to separate pools, transfers KV state and schedules each workload. EPD also separates image/video encoding, whose outputs still feed the language model. Gains come from reduced interference and independent resource ratios, not free compute.
T_transfer ≳ state_bytes / effective_link_bandwidth
Examples and evidence
Teaching lower bound: 1 GiB over an effective 25 GiB/s link takes at least about 40ms, before queueing and coordination. If avoided interference is smaller than added costs, separation may lose. Reevaluate arrival rates, modality, lengths, hardware and SLOs.
New constraints
Low load, short requests or slow networks may favor colocation. Separate pools can become imbalanced. The September 9, 2026 EPD source is a conditional technical analysis, not a mandate to split all multimodal services into three pools.
Primary sources
- DistServe: Disaggregating Prefill and Decoding · 2024-01-17 · Checked 2026-10-21
- Introducing NVIDIA Dynamo · 2025-03-18 · Checked 2026-10-21
- When to Use Encode-Prefill-Decode Disaggregation · 2026-09-09 · Checked 2026-10-21
Manual browser reading records and interactive calculators are available with JavaScript.
Keep exploring
S05 · How does continuous batching balance throughput and latency?