1. GPU capacity, autoscaling, and cost

GPU capacity planning for LLM serving decides how many model replicas to run, what each token costs, and how the replica count changes with traffic. The principle is short: measure how many requests one replica handles while still meeting the latency target, divide peak traffic by that number and add headroom. Then price the GPU-hours, and scale on signals that show waiting work rather than busy hardware.

Use it whenever you self-host a model against a stated traffic and latency target, explain a GPU bill, or serve traffic that swings by hour. Skip it in two cases. For low or spiky volume, a hosted API's per-token price usually beats any GPU you keep warm, a comparison covered in LLM Engineering 101. And if one replica misses the target with no load on it, more replicas will not help: fix batching, parallelism, or precision first.

The running example: a company self-hosts a 70B-parameter open-weight model for an internal chat assistant plus a nightly summarization job, on servers with eight 80GB GPUs. Weights in 16-bit take about 140GB, so one replica (one running copy of the model) is one 8-GPU server. Prompts run about 2,000 tokens and answers 300. Peak load is 50 requests per second, with TTFT (time to first token) under 1s and an assumed TPOT (time per output token) of 50ms.

The interviewer's question is how many GPUs that takes, what a million tokens costs, and how to avoid paying for peak all day. It comes at the end of most LLM serving rounds as a five-to-ten-minute quantitative check. A mid-level answer gets the replica arithmetic and the cost formula right. A senior answer also picks scaling signals, does the cold-start math, and keeps the nightly batch off the interactive budget. The chapters follow the order those follow-ups usually arrive in.