Skip to content
FiftyCalc Free calculators for all 50 states

A/B Test Calculator

Test cases replayed : 3 of 3 match · Methodology · No outside source listed

Enter the visitors and conversions of each version to compare their conversion rates.

Result

Sample figures

Two-tailed p-value—

Conversion rate A —

  • Conversion rate B—
  • Relative change, B vs A—
  • z statistic—
  • Decision at α—

Estimate for planning only, not tax, legal, financial or medical advice. This calculator uses the method explained on this page.

A/B Test Calculator

Version A: visitors: enter 1 or more. Prefilled example: 5,000.

Version A: conversions: enter 0 or more. Prefilled example: 200.

Version B: visitors: enter 1 or more. Prefilled example: 5,000.

Version B: conversions: enter 0 or more. Prefilled example: 250.

Significance level α: enter a number from 0 to 1. Prefilled example: 0.05.

How to use this calculator

Fill in the fields in this order; each step takes you to its field.

  1. Version A: visitors: enter 1 or more.
  2. Version A: conversions: enter 0 or more.
  3. Version B: visitors: enter 1 or more.
  4. Version B: conversions: enter 0 or more.
  5. Significance level α: enter a number from 0 to 1.
  6. Read the result: Two-tailed p-value; the panel also shows Conversion rate A, Conversion rate B, Relative change, B vs A, z statistic and Decision at α.

How this calculator works

Two-proportion z-test: z = (p_B − p_A) ÷ √(p̂ (1 − p̂) (1/n_A + 1/n_B)), where p_A and p_B are the conversion rates and p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B) is the pooled rate.

The two-tailed p-value is 2 × (1 − Φ(|z|)). It assumes visitors are independent and each version got enough conversions (at least about 10).

The formula

  1. Z=cb÷nb−ca÷naca+cbna+nb×(1−ca+cbna+nb)×(1na+1nb)
  2. P=2×(1−normcdf⁡(|Z|))
Symbols used in the formula
SymbolMeaning
naVersion A: visitors
caVersion A: conversions
nbVersion B: visitors
cbVersion B: conversions
Zz statistic
PTwo-tailed p-value
normcdf(x, m, s)the probability that a Gaussian variable with mean m and standard deviation s is at most x (mean zero and standard deviation one without m and s).

Worked example

Version A: visitors5,000
Version A: conversions200
Version B: visitors5,000
Version B: conversions250
Significance level α0.05
Conversion rate A4%
Conversion rate B5%
Relative change, B vs A25%
z statistic2.4119
Decision at αSignificant: p-value below α
  1. Z=250÷5,000−200÷5,000200+2505,000+5,000×(1−200+2505,000+5,000)×(15,000+15,000)=2.41192
  2. P=2×(1−normcdf⁡(|2.41192|))=0.015869

Two-tailed p-value 0.0159

Each step uses the unrounded value of the step before it; figures on this page are rounded for display.

Frequently asked questions

Is 4% vs 5% conversion on 5,000 visitors each significant?

Yes at α = 0.05: z = 2.4119 and the p-value is 0.0159.

Can I stop the test as soon as it becomes significant?

Checking repeatedly and stopping at the first significant result inflates false positives. Fix the sample size in advance, or use a method designed for sequential testing.

Questions answered

Related calculators