LLM inference is running a trained language model to answer a request. The server first processes the whole prompt in one parallel pass, called prefill, then loops, producing one answer token (a word or word piece the model reads and writes) per pass, called decode. Each decode pass must read every model weight out of GPU memory to produce a single token, so for a single stream, memory bandwidth sets the speed, not arithmetic.

Our running example is a company self-hosting a 70B-parameter open-weight model (140GB of 16-bit weights) on servers with eight 80GB GPUs, for an internal chat assistant and a nightly batch summarization job. Chat prompts run about 2,000 tokens and answers 300. Peak load is 50 requests per second, and the first token should appear in under 1s.

Use this view when asked why an LLM endpoint is slow or which metrics to track. Skip it for a hosted API whose GPUs you never run. Skip it too for fitting requests in memory, which is KV (key-value) cache sizing.

This is the opening question of most inference rounds. A mid-level candidate names prefill and decode, explains why decode is memory-bound, and defines time to first token and time per output token. A senior candidate does the bytes-per-token arithmetic unprompted, links the metrics with one formula, explains goodput, and picks a benchmark's operating point against an SLO (service-level objective). The expected depth is arithmetic, not GPU kernels (the programs that run on the GPU).