Why A B Tests Lie

When you keep checking an A/B test while it is running, you inflate the chance of declaring a winner that does not really exist. The video puts it as “the more you look, the higher the chance something looks significant by luck alone,” and that is the core of why “peeking” makes the numbers lie. The fix is to decide your sample size before a single visitor sees a variant, run the test until that size is reached, and then look once.

Why peeking inflates false positives

Every time you compare two groups, there is a small probability of seeing a difference by chance even when the variants are identical. By convention, that probability is set to 5 percent for a single comparison, often written as p < 0.05. If you perform that comparison ten times on the same data as it accumulates, each peek is an independent chance to be fooled, and the overall false-positive rate climbs well past 5 percent. With twenty peeks, the probability of at least one false “win” can exceed 30 percent, even though no real improvement exists. This is the same trap that catches teams who run many experiments in parallel: among twenty concurrent tests, roughly one will look like a winner on paper, and shipping every “winner” slowly degrades the product.

What “statistical power” actually means

Power is the probability that your test will detect a real effect of a given size, if one exists. It depends on three things: how big the effect is, how much the metric varies between users, and how many users you put into each arm. A common target is 80 percent power at a 5 percent significance level. Suppose your historical conversion rate is 5 percent with a standard deviation you have measured, and you want to detect a lift to 5.5 percent, a relative improvement of 10 percent. Plugging those numbers into a power calculation tells you the sample size per variant you need. If you only have half that sample when you peek and see a “win,” the result is roughly as trustworthy as a coin flip dressed up in a chart.

Common misunderstandings

• “The p-value is low, so the result is real.” A low p-value from a peeked-at test only tells you the data, as observed so far, are unlikely under the null. It says nothing about how likely you are to be wrong across the repeated checks you have already performed.
• “Stopping early is fine if the result is big.” Large early swings shrink as more data arrives. Stopping on a big number guarantees that you are conditioning on noise.
• “Bayesian methods let you peek freely.” Bayesian models with flat or diffuse priors still let you accumulate false certainty when updated many times on the same data. The discipline of committing to a stopping rule still applies.

When the standard advice does not apply

Sequential testing designs, such as always-valid p-values or group-sequential boundaries, do permit peeking while controlling error rates, but only if you planned the peeks in advance and use the right adjustment. Peeking ad hoc, by contrast, has no such protection. Likewise, if a test is monitoring a metric that drifts for business reasons, such as a holiday season, even a pre-planned sample size can mislead you, because the underlying baseline moved during the experiment.

The practical takeaway is simple: write down the sample size and the analysis plan before traffic is split, and treat the test as a sealed envelope you open only once. That is what makes an A/B test trustworthy, and it is the only setting in which the p-value you report means what most people assume it means.