1. Serving many models and fine-tuned variants

Multi-LoRA serving keeps one copy of a base model on the GPU and lets many fine-tuned variants share it. Each variant adds only a small LoRA (low-rank adaptation) adapter: two thin matrices whose product is added to a base layer's output. The server loads many adapters next to the base and runs a batch in which each request picks its own adapter, while the expensive base math happens once for everyone. How adapters get trained is out of scope here, and this tutorial starts once they exist.

Use it when you have many fine-tunes of the same base and none of them can fill a GPU alone, such as one per internal team. Don't use it when one variant carries most of the traffic: merge that adapter into the weights and give it dedicated replicas. Full fine-tunes, fine-tunes of different bases, and fleets of separate small models cannot share a base. Pack them onto GPUs instead, and give each tenant that needs hard isolation its own hardware partition of a GPU, or its own whole GPU.

"How would you serve 100 fine-tuned variants without 100 deployments?" is an optional topic. It gets probed deeply in model-hosting and internal ML platform rounds and shows up elsewhere as a passing follow-up. A mid-level answer says "share one base model, load adapters" and can size an adapter. A senior answer walks a mixed-adapter decode step, prices adapter swaps across memory tiers, routes by adapter without hot-spotting a replica, and knows what MIG (Multi-Instance GPU) and MPS (Multi-Process Service) do and do not isolate.

We use the course's running example: a company self-hosts a 70B-parameter open-weight model on servers with eight 80GB GPUs. Weights in 16-bit take about 140GB, a typical request is a 2k-token prompt plus a 300-token answer, and peak load is 50 requests per second. For this topic, the company also keeps 100 LoRA fine-tunes of that 70B, one per internal team.