Almost every product team ships changes through experiments, and almost every data science, analytics, and ML interview asks how you would test one. Saying "we'll A/B test it" is the easy part. The hard part is choosing what to randomize and measure, knowing how many users you need, telling a real lift from noise, and noticing when the result cannot be trusted. This course covers experiment design and statistical inference, one question at a time.

Each section takes one question interviewers actually ask about experiments and answers it as a standalone topic, with a small product example and the formulas you need to work it out.

What You Will Learn

• How to turn a product change into a testable experiment with the right randomization unit
• How to choose a primary metric, guardrails, and surrogates for slow outcomes
• How to tell a real difference from noise with hypothesis tests and confidence intervals
• How to size a test from the minimum detectable effect and turn it into a duration
• How to handle ratio metrics and cut variance with pre-experiment data
• What to check before trusting a result, from sample ratio mismatch to novelty effects
• Why peeking and many comparisons inflate false positives, and how to control them
• When users affect each other, and which cluster or switchback design fixes it
• How to make the call when metrics or segments disagree
• Optional: estimating impact when you cannot randomize
• Optional: Bayesian analysis with probability to beat control and expected loss
• Optional: running many experiments on the same users without them interfering

How This Course Works

Each section starts with the obvious approach, such as comparing two averages, stopping when the number looks good, or randomizing by user. Then it shows the concrete case where that approach gives the wrong answer, and adds the next idea only when that case calls for it.

Formulas appear wherever a concept is defined by one, always next to a worked number, and never with a derivation. Launching ML models through shadow mode and canaries is covered in Machine Learning System Design 101, not here.

Who This Course Is For

This course is for data scientists, data analysts, and ML or backend engineers preparing for experimentation questions in interviews. You should be comfortable with means, variances, and the normal distribution. You don't need a statistics degree.

This is not a course on building an experimentation platform, and it does not go into causal inference research methods.