1. Offline evaluation

You trained a model, and the interviewer asks the obvious next question: "How do you know it's good before you ship it?" Offline evaluation is the answer. You score the trained model on held-out historical data, data it never trained on, and use those numbers to decide whether it deserves a slot in a real experiment. It is the cheap filter before the expensive test, and its only job is to predict which model will win once real traffic arrives.

In a machine learning (ML) design round this usually takes 5-10 minutes, right after training. Mid-level candidates are expected to pick a metric that fits the task and set a decision threshold from the cost of each error. Senior candidates are also expected to design a split that mimics the future, spot leakage, and explain why an offline win can still lose online. Reporting only accuracy or AUC (area under the curve) on a shuffled test set is a common reason an answer gets rated below senior.

We will use one example throughout: the spam filter at a consumer email provider. About 5% of the mail that passes sender blocklists is spam, spam arrives in campaigns of thousands of near-identical messages, and sending a real message to the spam folder costs far more than letting one spam through.

You will come away able to plan offline evaluation for any new prompt, whether it is spam, fraud, or search: the metric, the threshold, the split, the slices, and what offline numbers cannot tell you. Running the live A/B test and sourcing the labels each have their own sections in this course.