A/B tests and their traps

An A/B test is an experiment: half the visitors see version A, half version B, and a metric chosen in advance is compared. Traps: small samples, peeking.

How to know that a change helped

You already have hypotheses, a counter and an idea of which numbers mislead. One question remains: how do you check that a change really helped? The most reliable way known is an experiment. On the web it is called an A/B test: half the visitors are shown version A (as it is), the other half version B (with the change), and then one metric chosen in advance is compared.

Where it comes from. The idea of a random split into groups is older than the internet: in the 1920s the statistician Ronald Fisher described randomized experiments on fields, and in medicine this is how drugs are tested (a placebo against a drug). On the web A/B tests became widespread in the 2000s, when large companies began to test versions of pages on millions of visitors. Google retired its free tool Optimize in September 2023, but the method remains, and the services for it vary: built into site builders, standalone or homemade.

Why randomness? If you show version B in the mornings and A in the evenings, you won’t know whether it was the version or the time of day. A random split (each person lands in one of the versions and always sees the same one) makes the groups alike in everything but the change. Then the difference is explained by the change.

The main idea: noise

People behave differently, and two identical groups will never give identical conversion. In the lesson on testing versions of text on readers you saw two identical texts “beat” each other on small numbers. That is called noise. An experiment has to tell a real effect from noise, and for that it needs two things: enough people and an honest stopping rule.

How many people are needed

The smaller the difference you want to notice, the more people you need. A rough rule for shares: per variant

n ≈ 16 × p × (1 − p) ÷ d²

where p is the current share (say 0.05 for 5%) and d is the minimum interesting difference as a fraction (0.01 if you want to notice a rise from 5% to 6%). The number 16 matches the usual thresholds: a false win in 5% of cases and an 80% chance of noticing a real difference.

An example. p = 0.05, d = 0.01: n ≈ 16 × 0.05 × 0.95 ÷ 0.0001 = 7,600 people per variant, that is 15,200 in all. If 600 people a day come to the page, that takes about 25 days. And if you have 40 people a month, such a test will never end. For comparison: to notice a difference between 5% and 25%, a few dozen people per variant are enough, but a single edit rarely brings an improvement like that.

This formula is rough, and for exact calculations there are sample size calculators (search for one by that name). More important is what it teaches: the sample is worked out before the test, not after.

Peeking

The most popular mistake. You launched a test, you look every day, and on the fifth day variant B is “significantly” better. You want to stop and rejoice. You can’t: the more often you look and the sooner you are ready to stop, the greater the chance of declaring a winner something that won by chance. Every look is a new chance to take noise for an effect. Evan Miller, in his article “How not to run an A/B test” (in the sources), shows this in numbers: a test stopped at the first “significance” is wrong several times more often than claimed.

Open the diagram. Set the number of visitors and run three hundred experiments with two identical variants. Without peeking there are about five percent “winners”. With ten peeks there are many more. Then switch on a real difference of one percentage point and see how often the test notices it at different sample sizes.

200 per variant1000 per variant3000 per variant7600 per variant15000 per variant

peek 10 times and stop at the first “win” variant B really converts at 6% instead of 5%

Run 300 experiments

A rule of thumb: to notice a difference of d percentage points at a share p, you need about 16·p·(1−p)/d² people per variant. For 5% against 6% that is about 7600. It is a rough estimate for the usual thresholds (5% false wins and an 80% chance of noticing). Everything in the diagram is simulated, there are no real people.

The safeguard: the duration and sample size are written down before launch, and the result is looked at once, at the end. If you badly want to keep watch, watch the health of the test (the number of people in the variants is about equal, nothing is broken), but make no decision.

Other traps

• Many metrics. If you look at twenty metrics at the five percent level, at least one will “significantly” change by chance in about two cases out of three (the multiple comparisons problem). So you choose one main metric before the test and keep the others only for control (to check that nothing important got worse).
• Seasonality and events. A week with a promotion is not like an ordinary one. Take full weeks (because behavior on weekdays and weekends differs) and write down everything that happened.
• Novelty. A new look draws attention just because it is new. A test must run long enough for people to get used to it.
• Changing things during the test. Editing the text, the price or the advertising in the middle of a test turns it into another test. Everything is frozen.
• A different mix. If one variant happens to get only mobile visitors, the difference is explained by the mix. The random split is made per person, not per visit.
• Goodhart’s law. The main metric must be what you really want (the lesson on metrics that mislead), with a control beside it.
• Ethics. Testing on people without telling them is common practice, but no excuse for deceit: don’t test dark patterns, and don’t hide the price.

When not to test

A test needs people. If a page has forty visitors a month, a test is pointless. Then the honest alternatives are:

• Big changes instead of small ones. A strong improvement (a clear promise, a short form) shows even on small numbers, and a change of a button’s color does not.
• Successive changes. Make changes one at a time, write down the date and compare periods (with caveats about seasonality).
• Qualitative research. Talk to five people from your audience and watch how they use the site. By Jakob Nielsen and Thomas Landauer’s estimate (1993), five people find about 85% of usability problems.
• A one-line survey. On the “Thank you” page: “What nearly stopped you before you sent the request?”

And one more honest reason not to test: when a change obviously fixes an error (the form didn’t send), you simply make it.

Check yourself: why can’t you stop a test as soon as the difference becomes “significant”?

Because every look at the result is a chance to see a random difference and take it for an effect. The more often you look, the more false wins. The duration and sample are written down in advance and looked at once.

Practice

Make a plan for one test.

1. The hypothesis (from the first lesson of the part) and a main metric about the main action from the brief.
2. The numbers: the current value (an assumption), the minimum interesting difference, the sample per variant by the rule, visitors a day, the duration.
3. The stopping rule and what we do not change.
4. Two traps and the safeguards.
5. Small traffic: what you’ll do in place of a test.

The check looks at the form of the plan. Whether your assumptions are sensible is for you to decide: if the duration comes out as half a year, don’t launch a test but take the plan for small traffic.