Batching runs many users' requests through the model in the same GPU step. The scheduler decides who is in that step and who waits. It works because a decode step reads all the model weights once no matter how many sequences it advances, so sharing the step multiplies throughput almost for free. Use it on any shared LLM endpoint with more than about 4 concurrent decoding users, and on offline jobs priced per GPU-hour. Skip it for single-user local deployments, and use plain dynamic batching for single-pass models such as embedding models.

"How do you batch requests of very different lengths?" comes up in most LLM serving rounds. A mid-level answer explains continuous batching and why bigger batches raise throughput but slow each user's tokens. A senior answer puts numbers on the step-time curve, fixes long-prompt interference with a token budget, weighs separate prefill and decode pools, and prices preemption.

This chapter assumes prefill vs. decode from The LLM Inference Workload and why KV memory caps concurrency from GPU Memory and the KV Cache.

The running example: a company self-hosts a 70B-parameter open-weight model for an internal chat assistant plus a nightly summarization job, on servers with eight 80GB GPUs. Weights in 16-bit take about 140GB. A typical prompt is 2,000 tokens and a typical answer 300. Peak load is 50 requests per second. Targets are TTFT (time to first token) under 1s and an assumed TPOT (time per output token) of 50ms. The KV (key-value) cache costs about 0.33MB per token.