A/B test significance calculator
Enter how many visitors saw each version and how many converted. The calculator compares the two conversion rates with a two-proportion z-test and tells you whether the difference is statistically significant at the confidence level you choose.
Runs in your browser: nothing you enter is sent anywhere.
How to use it
- Enter visitors and conversions for A and for B.
- Choose the confidence level and the test type.
- Read the p-value; significant means the p-value is below 1 − confidence.
How it works
- Rates: conversions ÷ visitors. Lift = (rate B − rate A) ÷ rate A.
- z = (rate B − rate A) ÷ √(p(1 − p)(1/nA + 1/nB)) with the pooled rate p.
- Two-sided p = 2 × (1 − Φ(|z|)); one-sided p = 1 − Φ(z), where Φ is the normal distribution.
- The interval uses the unpooled standard error and the z value for your confidence level.
Worked example
1,000 visitors each; A converts 100 (10%), B converts 130 (13%). The pooled rate is 11.5%, the standard error 0.01427 and z = 2.10. The two-sided p-value is 0.0355, below 0.05, so the result is significant at 95%. The lift is +30%.
Tips
- The test assumes independent visitors and enough conversions in each group (roughly 10 or more).
- Significant does not mean big: look at the interval for the size of the effect.
- Statistics here are estimates, not business or financial advice.
Frequently asked questions
What does the p-value mean?
It is the probability of seeing a difference at least this large if A and B really converted equally. A small value (below 0.05 at 95% confidence) is evidence against "no difference". It is not the probability that B is better.
How is the test calculated?
The pooled rate is (conversions A + B) ÷ (visitors A + B), the standard error is √(p(1 − p)(1/nA + 1/nB)) and z = (rate B − rate A) ÷ standard error. The p-value comes from the normal distribution.
Should I pick one-sided or two-sided?
Two-sided is the safe default: it detects B being better or worse. Choose one-sided only if you decided before the test that you care about B being better and would ignore a loss.
What is the sample size figure?
The visitors needed in each version to detect a difference as large as the one you observed, at your confidence level and chosen power. It is a planning figure; if the observed gap is a lucky fluctuation it will be too small.
Can I stop the test as soon as it turns significant?
Checking repeatedly and stopping at the first significant result inflates false positives. Decide the sample size first and read the result once.