1. Interference and network effects

"In a two-sided marketplace, why might a user-level A/B test give the wrong answer?" This follow-up usually arrives right after you have set up a clean test, and it asks whether your test measures what you think it does. The concept behind it is interference: one user's treatment changes another user's outcome. A standard A/B test assumes that never happens. In marketplaces and social products it happens all the time.

The fix is to move the randomization boundary, to friend clusters, whole cities, or time windows, so that users who interact land in the same arm. When users neither compete for scarce supply nor influence each other, as with a checkout-page redesign, an ordinary user-level test is fine.

Expect 5-10 minutes on it at delivery, ride-hailing, lodging, and social companies, and much less elsewhere. A mid-level answer names the broken assumption and says which way the result is biased. A senior answer also picks a design that fixes it, estimates what that design costs in statistical power, and says how to check a finished test for contamination.

The running example is a food delivery marketplace in 40 cities with about 6M weekly customers, roughly 150k per city. Couriers are almost fully booked at dinner peak (6-9 pm). The team tests a $2 delivery-fee discount with a user-level 50/50 split in one city, and treatment customers order about 10% more than control. Yet when the team later measures what happens if everyone gets the discount, orders rise only about 3%. This tutorial explains the gap between those two numbers and the designs that measure the right one.

Choosing a basic randomization unit (user, session, account) is covered in Experiment design. Here the question is what to do when no user-level unit is independent enough.