The KV cache is GPU memory that holds the attention keys and values each layer computed for every token a request has seen so far, so the model does not recompute them for every new token. GPU memory first holds the model weights. Whatever is left becomes KV cache space, and each running request takes KV memory in proportion to its token count. Leftover memory divided by memory per request is how many requests one server can run at once.

Reach for this arithmetic when an interviewer asks whether a model fits, how many GPUs it needs, or how many concurrent requests or how long a context one server can hold. Skip it for a small model on a big GPU with short prompts, where the cache never fills and the time is better spent on latency. Skip it too for questions about which request runs next or which replica gets it. Those are scheduling and routing questions.

Our running example is a company self-hosting a 70B-parameter open-weight model for an internal chat assistant plus a nightly batch summarization job, on servers with eight 80GB GPUs. Weights in 16-bit precision take about 140GB, a typical chat request is a 2,000-token prompt with a 300-token answer, peak load is 50 requests per second, and the first token must arrive in under 1s. Every number in this tutorial comes from that setup.

"Does it fit, and how many requests at once?" is the most common sizing question in LLM serving rounds, and it usually comes right after prefill vs. decode. A mid-level candidate is expected to write the memory budget and the per-token KV formula with real numbers. A senior candidate goes on to defend paged allocation, prefix reuse, and what happens when the cache fills, with numbers attached. The chapters follow the order those questions usually arrive in.