1. Peeking and multiple testing

Your checkout test is planned for 14 days, but the product manager (PM) opens the dashboard every morning. On day 5 the one-page checkout shows p = 0.03, and the PM asks whether the team can stop now and ship. The answer is no: a p-value below 0.05 caps false wins at 5% only when you look once, at one planned comparison. Every extra look over time (peeking) and every extra metric, variant, or segment you test (multiplicity) gives random noise another chance to cross the line.

Interviewers use this prompt constantly, usually in the analysis and monitoring part of an experimentation round, and it rarely takes more than 5 to 10 minutes. A mid-level answer explains why daily peeking and a 20-metric deck inflate false wins, puts a rough number on it, and states the fix: a fixed horizon and one primary metric chosen before launch. A senior answer goes further and picks a tool: a sequential design when the team really needs early stops, and Bonferroni or Benjamini-Hochberg depending on which error matters.

You should already be comfortable with the two-proportion z-test and what alpha means, from Hypothesis Testing. You should also take the 14-day horizon as given. Everything here stays inside one experiment's error control, using the same checkout test: control converts 10.0%, about 2.5k users per arm arrive each day, and the results deck tracks 10 metrics, a second variant, and a handful of segments.