1. Designing an LLM serving system

"Design an LLM inference service for this product." An LLM serving system is the service that takes chat or batch requests, runs them on GPU replicas of a model, and streams the answers back within a latency target. Requests pass a gateway and a router into a queue in front of engine replicas. Each replica holds the model weights plus a KV (key-value) cache, the per-request attention memory. Designing one means sizing memory and throughput from the workload first, then choosing how to batch, split, route, scale, and recover.

Use this full answer when the interviewer hands you a product and a traffic target, or when a team self-hosts an open-weight model and has to turn a forecast into GPUs. Skip it in two cases. If the product calls a hosted model API, design the application side instead: caching, prompt cost, fallbacks. If the question is about one mechanism, such as "does it fit?", answer that mechanism directly.

This prompt usually opens the round and takes up all of it: 45 to 60 minutes, with request flow, failures, rollout, and mixed traffic arriving as follow-ups. A mid-level candidate is expected to walk a sensible order and get the memory math right. A senior candidate drives every step with a number, picks the deep dives, and handles the failure and rollout questions without being led.

We'll use the course running example throughout: a company self-hosting a 70B-parameter open-weight model for an internal chat assistant plus a nightly batch summarization job, on servers with eight 80GB GPUs. Weights at 16-bit take about 140GB. A typical chat request is a 2k-token prompt with a 300-token answer, peak is 50 requests per second, and TTFT (time to first token) must stay under 1s.