1. Quantization

Quantization stores a trained model's numbers in fewer bits, for example 8 or 4 bits per weight instead of 16, so the model takes less GPU memory. It works by mapping each block of 16-bit values onto a small grid of 8-bit or 4-bit values plus one scale factor that says how big a grid step is. The GPU reads the small values and either converts them back or computes with them directly. Every generated token then moves fewer bytes out of memory, at the cost of a little rounding error in every number.

Reach for it in three cases: the 16-bit model does not fit or leaves too little room for the KV (key-value) cache, decode is memory-bound and you want more tokens per second from each GPU, or you want more concurrent requests from the same memory. Skip it when the model already fits with plenty of headroom and your quality budget is tight: keep 16-bit and add replicas. If the workload is mostly long prompts, quantizing only the weights will not speed it up. Look at 8-bit compute or more GPUs.

In an LLM serving round this is the "make it smaller" prompt: how would you cut memory and cost by lowering precision, and how do you know quality held? It usually takes a few minutes inside a larger serving design. A mid-level answer names INT4 or INT8 weight-only versus FP8 and says quality must be checked before and after. A senior answer explains why weight-only quantization helps decode but not prefill, quantizes the KV cache to fit more requests, and names outlier layers and long-context regressions as where quality breaks.

We use the course's running example throughout: a 70B-parameter open-weight model serving an internal chat assistant and a nightly batch summarization job on servers with eight 80GB GPUs. At 16 bits the weights take about 140GB, a typical request is a 2k-token prompt with a 300-token answer, peak load is 50 requests per second, and time to first token must stay under 1s. This tutorial covers quantizing an already-trained model for serving. Quantization-aware training and distillation change the model itself and are out of scope.