How does continuous batching balance throughput and latency?

The bottleneck

Static request batches can make short requests wait for long ones and leave completed slots unfilled. Orca’s iteration-level scheduling admits and removes requests at iteration boundaries, improving utilization while creating finer scheduling decisions.

The mechanism

Each iteration is selected from pending inputs and decoding requests under token and memory budgets. Heavy prefill work can interfere with latency-sensitive decode. Chunked prefill can complement this scheduling, but is not an automatic property of every continuous-batching implementation.

TTFT = first_token_time − arrival; TPOT ≈ (last − first)/(output_tokens − 1)

Examples and evidence

Teaching schedule: iteration 1 has A/B/C; after B finishes, iteration 2 has A/D/C; after A finishes, iteration 3 has E/D/C. Replacing slots does not imply equal request costs. This TPOT estimate requires at least two output tokens; means can hide tail latency.

New constraints

High throughput does not equal good interaction. Report TTFT, inter-token latency, tail percentiles and SLO-compliant throughput alongside arrival rate and length distributions. Aggressive batching may increase throughput while slowing every user.