A/B Test Calculator
Test cases replayed : 3 of 3 match · Methodology · No outside source listed
Enter the visitors and conversions of each version to compare their conversion rates.
Result
Two-tailed p-value
Conversion rate A
- Conversion rate B
- Relative change, B vs A
- z statistic
- Decision at α
Estimate for planning only, not tax, legal, financial or medical advice. This calculator uses the method explained on this page.
This browser did not allow copying. The sentence is in the box below, already selected: copy it with your keyboard or the menu.
How to use this calculator
Fill in the fields in this order; each step takes you to its field.
- Version A: visitors: enter 1 or more.
- Version A: conversions: enter 0 or more.
- Version B: visitors: enter 1 or more.
- Version B: conversions: enter 0 or more.
- Significance level α: enter a number from 0 to 1.
- Read the result: Two-tailed p-value; the panel also shows Conversion rate A, Conversion rate B, Relative change, B vs A, z statistic and Decision at α.
How this calculator works
Two-proportion z-test: z = (p_B − p_A) ÷ √(p̂ (1 − p̂) (1/n_A + 1/n_B)), where p_A and p_B are the conversion rates and p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B) is the pooled rate.
The two-tailed p-value is 2 × (1 − Φ(|z|)). It assumes visitors are independent and each version got enough conversions (at least about 10).
The formula
| Symbol | Meaning |
|---|---|
| Version A: visitors | |
| Version A: conversions | |
| Version B: visitors | |
| Version B: conversions | |
| z statistic | |
| Two-tailed p-value | |
normcdf(x, m, s) | the probability that a Gaussian variable with mean m and standard deviation s is at most x (mean zero and standard deviation one without m and s). |
Worked example
| Version A: visitors | 5,000 |
|---|---|
| Version A: conversions | 200 |
| Version B: visitors | 5,000 |
| Version B: conversions | 250 |
| Significance level α | 0.05 |
| Conversion rate A | 4% |
|---|---|
| Conversion rate B | 5% |
| Relative change, B vs A | 25% |
| z statistic | 2.4119 |
| Decision at α | Significant: p-value below α |
Two-tailed p-value 0.0159
Each step uses the unrounded value of the step before it; figures on this page are rounded for display.
Frequently asked questions
Is 4% vs 5% conversion on 5,000 visitors each significant?
Yes at α = 0.05: z = 2.4119 and the p-value is 0.0159.
Can I stop the test as soon as it becomes significant?
Checking repeatedly and stopping at the first significant result inflates false positives. Fix the sample size in advance, or use a method designed for sequential testing.