1. Online experimentation and rollout

Your new model won offline, and the interviewer asks: "How do you launch it and prove it helps?" Online experimentation and rollout is the answer. You expose the new model to live traffic in stages, and each stage answers one question. Shadow mode asks whether it runs correctly. A canary asks whether it is safe for real users. An A/B test asks whether it actually moves the business metric.

Before any of it starts, you write down what result ships the model and what result rolls it back. Not every release needs all three stages. A routine retrain of the same model on fresh data usually passes an offline check plus shadow and canary, and the full A/B test is kept for changes users would notice, such as a new ranker.

This is usually the last five minutes of an ML (machine learning) design round, and weak candidates skip it entirely, which is itself a signal. Mid-level candidates are expected to order the stages and set up a user-randomized A/B test with one primary metric and guardrails. Senior candidates also catch the randomization and early-stopping traps, use interleaving to screen rankers cheaply, and state ship and rollback rules before anyone looks at a dashboard.

We will use one example throughout: restaurant search in a food-delivery app. A user types "pizza" and a ranker orders the matching restaurants. Ranker v1 is live. Candidate v2 beat it offline, with NDCG@10 (normalized discounted cumulative gain over the top 10 results) of 0.431 against 0.412. About 2M users search each week, and the business metric is orders per user per week.

By the end you can lay out the launch plan for any model, from ads ranking to fraud scoring: the stages, the test unit, the metrics, the duration, and the rules. Offline metrics and experiment statistics each have their own sections in this course.