1. Operating a GPU inference fleet

GPU fleet operations is the day-to-day work of keeping a self-hosted fleet of model replicas inside its SLO (service-level objective), the latency target users are promised, while GPUs fail and GPU supply runs short. The basic principle: treat each multi-GPU replica as the unit that fails, and test it with a small timed generation. Pull failing or slow replicas out of rotation and replace them from spare capacity. Page a human on user-facing latency and errors, and spread capacity across regions and purchase types so a shortage in one place has a fallback.

You need this once you run your own GPU servers for a model with a latency target and the fleet is large enough that some GPU fails most weeks, or once you serve from a region where GPUs cannot always be bought on demand. You do not need it when you call a hosted model API. There the provider runs the fleet, and your job is to watch the API's latency and errors and keep a fallback provider. A few GPUs do not need it either: a restart and a manual replacement are enough until failures become routine.

The interview question sounds like this: "Your LLM service runs on hundreds of GPUs. GPUs fail, regions run out of capacity, and latency drifts. How do you keep it healthy?" Inference-platform, model-hosting, and ML infra SRE (site reliability engineering) rounds ask it as follow-up after the serving design. A mid-level answer names the failure modes, separates liveness from readiness, and lists sensible metrics. A senior answer adds the failure arithmetic, drain mechanics for streaming requests, burn-rate paging, and an agreed order of moves for a GPU shortage.

The running example is the course's company serving a 70B open-weight model for an internal chat assistant plus a nightly batch job, on eight-GPU 80GB servers. One replica is one 8-GPU server at TP=8 (tensor parallelism across eight GPUs), sustaining about 5 rps at p95 TTFT (time to first token) under 1s (GPU Capacity, Autoscaling, and Cost). The chat service runs in three regions, each sized for half the 50 rps peak: 25 / 5 = 5 replicas × 1.3 headroom ≈ 7 servers, 21 servers and 168 GPUs in all.