1. Routing and admission control

Routing picks which model replica serves each LLM request. Admission control decides whether the fleet accepts that request at all, or delays, degrades, or rejects it when the fleet is full. The basic principle is that LLM requests differ in cost by 10x or more depending on prompt and output length. So the router balances load by work measured in tokens rather than by request counts, and it steers repeat prefixes to the replica that already cached them. The front door counts tokens per tenant and stops admitting work before queues grow long enough to push latency past its target.

You need this for any fleet of two or more LLM replicas behind one endpoint, and for any shared multi-tenant endpoint where one team's burst can hurt everyone else. You don't need it for a single replica (set a queue limit on the engine and stop), or for uniform single-pass models such as embedding models, where plain least connections and request-count rate limits are enough.

"How do you load balance LLM requests, and why does round robin fail?" opens most LLM serving rounds. A mid-level answer routes on tokens and bounds the queue. A senior answer trades cache affinity against hot spots with a load bound, orders the overload responses for a mixed workload, and charges tenants tokens with a reservation and a refund. This tutorial assumes prefill and decode from The LLM Inference Workload and continuous batching from Batching and Scheduling, and it stops at the replica boundary.

The running example: a company self-hosts a 70B-parameter open-weight model for an internal chat assistant plus a nightly summarization job, on servers with eight 80GB GPUs. A typical chat prompt is 2k tokens and a typical answer 300. Peak load is 50 requests per second across 10 replicas, one 8-GPU server each, so about 5 requests per second per replica. The target is TTFT (time to first token) under 1s. Tenants are internal teams sharing the chat endpoint.