1. Model serving

A trained model is a function that turns feature values into a score, but a score only helps if it reaches the product while the user is still waiting. Model serving is the part of the system that gets it there: it decides when scores are computed, ahead of time or per request, what path a request takes from feature lookup through the model to the final list, and what the system returns when part of that path is slow. In a machine learning (ML) design round it answers the question that comes right after training and evaluation: how does a prediction reach the user within the latency budget?

The one idea behind every serving choice is to compute each part when its inputs change. Precompute when you can list the inputs ahead of time and staleness is fine, such as a weekly email digest; then no model runs on the request path at all. Put the model on the request path only when the score must see this request, and run the heavy model only on a short list. How the features themselves are computed and how a new model version is rolled out are separate sections of this course. Here the features and the model are given, and the question is how they get to a request in time.

We'll use one example throughout: the home feed of a second-hand marketplace app. It has 30M active listings and 40M registered users, of whom about 8M open the app on a given day. At peak the feed handles 20k requests per second. The feed API must answer in 200ms at p99, and it gives the ML scoring path 120ms of that.

Serving usually takes 5-10 minutes of an ML design round, longer when the prompt is a ranking problem. A mid-level answer explains the choice of batch, online, or hybrid inference, names the request stages, and gives the model call a timeout and fallback. A senior-level answer also sets the overall p99 budget, divides it among stages, sizes the candidate set at each stage, and orders the fallbacks.