1. Speculative decoding

Speculative decoding lets a small, fast model guess the next few tokens and has the big model check all of those guesses in one step. Every guess the big model agrees with is a token gained without a separate big-model step. At the first guess it disagrees with, the big model's own token replaces it, so one step can produce several tokens and the answer is the one the big model would have written alone.

It works because, at small batch sizes, a decode step spends its time reading the weights, not doing math, so checking five tokens costs about the same as generating one. Use it for interactive traffic at small or moderate batch and for predictable output such as code or summaries that copy from the prompt. Skip it for large-batch offline jobs that already keep the GPU busy, where a bigger batch is the simpler win, and for creative, high-temperature text (sampling with a lot of randomness), where plain decoding is simpler.

In a serving round it comes up after batching: batching raises total throughput but does not make one user's tokens arrive faster. It is an optional topic, probed in depth mainly by inference-engine and model-hosting teams. A mid-level answer explains draft-and-verify and says output quality does not change. A senior answer computes the speedup from acceptance rate and draft cost, gives the accept rule, picks a draft source, and explains why the gain fades at large batch.

We use the course's running example: a 70B-parameter open-weight model serving an internal chat assistant and a nightly batch summarization job on servers with eight 80GB GPUs. At 16 bits the weights take about 140GB, a typical request is a 2k-token prompt with a 300-token answer, and peak load is 50 requests per second. Each replica spans two GPUs and runs a handful of concurrent decodes off-peak and tens at peak.