01The formula
| To get | Calculate |
|---|---|
| Conversion rate | Conversions ÷ visitors |
| Relative lift | (Rate B − rate A) ÷ rate A |
| Test used | Two-proportion z-test, two-sided, pooled standard error |
| Significant at 95% | P-value below 0.05 |
02What significant means
A significant result means that if the two versions were truly the same, a gap this large would show up less than 5% of the time by chance. It doesn't mean there's a 95% chance B is better, and it says nothing about whether the lift is big enough to matter.
03How tests go wrong
- Stopping when it turns green. If you check every day and stop the first time it's significant, you'll find winners that aren't there. Decide the sample size first.
- Too small a sample. Small tests only detect huge differences. A flat result on a small sample means you don't know yet.
- Trusting the short term. At Shopify I ran 40+ structured experiments a quarter. When we measured long enough, about 40% of the experiments that showed a positive short-term result had no long-term effect.
- Measuring the wrong event. A test that wins on clicks or signups can lose on paying customers.
04When an A/B test can't answer the question
An A/B test compares two versions of something you control. It can't tell you whether a channel adds customers who wouldn't have come anyway. For that you need a holdout: switch the channel off in matched markets and compare. I cover the design in incrementality testing and a worked example in is branded search incremental. For the creative side, see my ad creative testing framework.
FAQQuestions people ask
- How do you calculate statistical significance for an A/B test?
- Compare the two conversion rates with a two-proportion z-test. The difference between the rates is divided by its standard error to get a z-score, and the z-score gives a p-value. A p-value below 0.05 is significant at 95% confidence.
- What does a p-value of 0.05 mean?
- If there were truly no difference between the versions, you'd see a gap at least this large about 5% of the time by chance alone. It isn't the probability that your variant is better.
- How long should I run an A/B test?
- Until you reach the sample size you set before starting, and for at least one full week so every day of the week is covered. Stopping early because the result looks good inflates false positives.
- My test isn't significant. Does that mean there's no difference?
- No. It means the test couldn't tell. With a small sample, real differences often go undetected. Either run it longer or accept that the effect, if any, is smaller than your test could see.
MoreMore free tools
CPM calculatorROAS calculatorCAC calculatorLTV calculatorCPA calculatorPipeline calculatorMarketing ROI calculatorGoogle Ads budget calculatorUTM builderllms.txt generatorAEO checker