1. Model parallelism for inference

Your interviewer sets up the design: a 70B-parameter model serving an internal chat assistant and a nightly batch summarization job, on servers with eight 80GB GPUs. Then comes the usual follow-up: "The model doesn't fit on one GPU. What do you do?" Model parallelism is the answer. You split one model across several GPUs so that together they hold and run it as a single copy, called a replica. Each GPU keeps part of the weights, and the GPUs exchange intermediate results. How you cut the model decides how much data they exchange and how often.

You need a split when the 16-bit weights alone (140GB for the 70B model) exceed one 80GB GPU, when the model fits but leaves too little memory for the key-value (KV) cache to batch enough requests, or when you serve a mixture-of-experts model whose total weights are far larger than the part any one token uses. You don't need one when the model fits on a GPU with room to spare: run one copy per GPU and add replicas. And if 8-bit or 4-bit weights would fit on fewer GPUs at acceptable quality, try Quantization before you split.

Interviewers usually give this 5 to 10 minutes. A mid-level answer names tensor and pipeline parallelism correctly and knows why tensor parallelism stays inside one server. A senior answer brings the numbers (link bandwidth, the pipeline's idle fraction), handles expert parallelism for mixture-of-experts models, and weighs bigger replicas against more replicas for a given workload.

Every chapter reuses the course's running example: 140GB of weights, a typical prompt of 2k tokens and answer of 300 tokens, a peak of 50 requests per second, and time to first token under 1s. Training splits the model in other ways for other reasons and is out of scope here. The chapters move from whether to split, through the three ways to cut, to how big each replica should be.