1. Validity checks and common pitfalls

Your home-goods marketplace tests a redesigned search results page with larger photos and price badges. Two weeks in, the dashboard says search-to-purchase conversion is way up, and the interviewer asks the follow-up that decides the round: before you believe it, what would you check?

Validity checks are the answer. They test whether each arm holds the users the design said it would, whether both arms were measured the same way, and whether the test window shows the effect you'd actually get after launch. Passing them makes the comparison fair, not significant: whether the lift is real is still a hypothesis-testing question. And not every check applies to every test. A backend change users cannot see needs no novelty analysis.

This question shows up right after the analysis step of most experimentation rounds, usually as 5 to 10 minutes of follow-ups. For a mid-level role, the bar is running a chi-square check on the arm counts and knowing what a clean A/A test (both arms get the identical experience) looks like. For a senior role, interviewers also listen for carryover from earlier tests, novelty effects, mix shift, and a firm refusal to "fix" a broken split by reweighting.

The tutorial assumes you know how users are hashed into arms with a salt, from Experiment Design, and how to read a p-value against alpha, from Hypothesis Testing. It covers the checks that decide whether the comparison is fair. Whether a fair difference is significant, and what happens when you look at the dashboard every day, are separate sections of this course.