Serving large language models is now one of the most expensive jobs in infrastructure, and a hosted API hides the hard parts. Running models yourself means knowing how much GPU memory each request needs, how to keep expensive GPUs busy, how many of them you need, and how to stay fast when traffic spikes. This course teaches LLM inference infrastructure at the depth AI infrastructure interviews test, with the numbers behind every choice.

Most sections follow one deployment: a company self-hosting a 70B-parameter open-weight model for an internal chat assistant and a nightly batch job, on servers with eight 80 GB GPUs. Each section works out what its topic does to that deployment's memory, latency, throughput, or cost.

What You Will Learn

• Why LLM inference is slow and expensive: prefill vs decode, why decode is limited by memory bandwidth, and how to measure a server with time to first token, time per output token, and goodput
• How to budget GPU memory, size the KV cache, and fit more concurrent requests with paged allocation, prefix caching, and offload
• How continuous batching, chunked prefill, and separate prefill and decode pools raise throughput without stalling users
• How to split a model that does not fit on one GPU with tensor, pipeline, and expert parallelism, and when more replicas beat bigger ones
• How quantization cuts memory and cost, and how to check that quality held
• How to route requests across replicas for load and cache hits, and how admission control and token-based quotas protect latency under overload
• How to turn a traffic and latency target into a GPU count and a cost per million tokens, and how to autoscale around slow cold starts
• How to approach the "design an LLM inference service" round with a reusable framework, from workload and SLOs to failures and model rollouts
• Optional: how speculative decoding speeds up each user's tokens, and when it stops helping
• Optional: how to serve many fine-tuned variants and small models on shared GPUs
• Optional: how to detect and replace failing GPU replicas, and keep capacity when a region runs out of GPUs

How This Course Works

Each section starts with the simplest setup, such as one request at a time on one GPU in full precision. Then it shows the number that makes that setup too slow or too expensive, and adds the next technique only when that number calls for it.

Every mechanism comes with arithmetic on the running deployment, taught to the depth an interviewer probes: enough to choose and defend a technique, not to implement it. The last core section turns these pieces into a reusable way to answer an open serving design round; it uses short examples to show the framework rather than solving one product design end to end.

Who This Course Is For

This course is for infrastructure, ML platform, and backend engineers and SREs moving into AI infrastructure roles. You should be comfortable with distributed systems basics and capacity planning. You don't need GPU programming or deep learning research experience.

This is not a course on training infrastructure, writing GPU kernels, or a specific serving framework. Deciding whether to self-host at all is covered in LLM Engineering 101, and general SLO and capacity practice in SRE & Reliability Engineering 101.