"Walk me through how you'd set up an A/B test for this change." That question opens most experimentation rounds, and the answer is experiment design: the choices you make before launch that decide what the test can prove. You settle whether a test fits at all, write the hypothesis, decide who counts and when, pick the unit you randomize, assign units to control and treatment, and write the decision rule down before the first user arrives. Analysis cannot fix a test that assigned the wrong users or measured the wrong outcome.

The principle is that random assignment makes the change the only systematic difference between the two groups. A test fits when you have enough users, the change can be turned off, and withholding it is acceptable. When one of those fails, as with a 200-person beta or a public rebrand, use interviews or judgment instead.

Expect it in the first 5-10 minutes of a product data science, analytics, or machine learning (ML) case. A mid-level answer states a hypothesis with a direction, randomizes by user, and splits 50/50. A senior answer also says when not to test, triggers the analysis where the arms first differ, matches the analysis unit to the randomization unit, and fixes the decision rule in advance.

Choosing the metric and sizing the test are separate sections of this course. This one covers the online two-arm A/B setup, not classical design of experiments such as factorial designs or blocking.

The running example is an online grocery delivery app with about 2M weekly active users in about 1.6M household accounts, where one cart is shared by everyone in the household. The team wants to test an express checkout button on the cart page that skips the order-review step. About 500k users open the cart page in a typical week, about 30% of them check out, and the team hopes for +1.5 points.