1. Production monitoring and retraining

The model shipped, the canary passed, and the interviewer asks the question that closes most machine learning (ML) design rounds: "It's live. What do you watch, and when do you retrain?"

Production monitoring is the answer to the first half. You watch the live model at four layers so you catch the case that matters most: a model that still answers every request but has quietly become wrong. Retraining is the second half: deciding when a fresh model replaces the live one, and what it must prove before it does.

How much of this you need depends on how fast labels arrive. When the true outcome lands within minutes, as with ad clicks, quality metrics on fresh labels and a scheduled retrain cover most of it. Drift monitors, early proxy signals, and retraining triggers earn their cost when labels lag by weeks, as they do in lending.

This usually gets the last 3-5 minutes of a round, so its interview weight is moderate. Mid-level candidates are expected to name the layers with real signals and tell data drift from concept drift. Senior candidates are also expected to notice that the true label arrives weeks late, pick an earlier signal, set retraining triggers behind a promotion gate, and spot when the model's own decisions corrupt its training data. Saying "monitor accuracy and retrain weekly" is the classic weak answer, because accuracy is exactly the number you cannot see yet.

One example runs throughout: a buy-now-pay-later lender scoring checkout applications. About 400k applications arrive each week, the model gives each a probability of default, applications above 0.12 are declined, and about 70% are approved.

The true label is whether the borrower defaults within 90 days, and installments are due every 14 days. The live model is v3, trained on last year's January–June applications and launched in the autumn. You will leave able to write a monitoring and retraining plan for any live model, from fraud scoring to a feed ranker. Running the launch experiment and running a training job are separate sections of this course.