A/B Test Statistical Significance — Z-Scores, P-Values, and Sample Size
An A/B test compares two variants of a page, email, or feature to determine which performs better on a target metric (conversion rate, click-through rate, revenue per visitor). But observed differences can arise from random variation — statistical significance testing quantifies the probability that the difference is real. This calculator applies the two-proportion Z-test: the standard method for comparing binary conversion rates between two independent groups.
Where p₁, p₂ are the observed conversion rates; n₁, n₂ are the sample sizes; and p̂ is the pooled conversion rate (total conversions / total visitors). Significance is declared at 95% confidence when |Z| > 1.96, and at 99% confidence when |Z| > 2.576.
The Two-Proportion Z-Test
The two-proportion Z-test is the standard method for A/B testing binary outcomes (converted vs. did not convert). It computes how many standard deviations the observed difference between variants is from zero — the Z-score. The Z-score is then mapped to a p-value using the standard normal distribution. A Z-score above 1.96 corresponds to p < 0.05 (95% confidence). Above 2.576 corresponds to p < 0.01 (99% confidence). The pooled proportion p̂ accounts for the fact that under the null hypothesis, both groups are drawn from the same underlying conversion rate.
Type I and Type II Errors
Statistical testing involves two kinds of errors. A Type I error (false positive) is declaring a winner when the variants are actually equivalent — controlled by the significance threshold (α). At 95% confidence, α = 0.05: you accept a 5% chance of a false positive. A Type II error (false negative) is failing to detect a real improvement — controlled by statistical power (1 − β). Standard practice targets 80% power (β = 0.20), meaning a 20% chance of missing a real effect of the specified size. Increasing sample size reduces both error rates simultaneously.
Sample Size and the MDE
The required sample size grows rapidly as the minimum detectable effect (MDE) shrinks. The formula scales as 1/MDE². Halving the MDE you want to detect quadruples the required sample size. For a 5% baseline conversion rate: detecting a 20% relative lift (to 6%) requires ~5,000 visitors per variant at 95%/80% power. Detecting a 5% relative lift (to 5.25%) requires ~80,000 per variant. Always calculate required sample size before starting a test — most A/B tests that fail to reach significance are underpowered, not evidence that variants are equal.
Statistical vs Practical Significance
Statistical significance tells you the effect is real; it does not tell you it matters. With a very large sample size, a 0.1% absolute conversion lift can be statistically significant at p < 0.001 — but if that lift generates $20/month in revenue, it may not be worth deploying. Always pair statistical significance with practical significance: does the measured effect size justify the implementation cost and ongoing maintenance? Report confidence intervals, not just p-values — a 95% CI of [+0.1%, +0.3%] tells a very different story than [+2%, +8%], even if both are 'significant'.
Frequently Asked Questions
What is a statistically significant result in an A/B test?
Statistical significance means the observed difference between variant A and variant B is unlikely to be due to random chance. The standard threshold is 95% confidence — meaning there is less than a 5% probability (p < 0.05) that the observed difference would occur if both variants performed identically. At this threshold, you are accepting a 1-in-20 chance of declaring a winner when none exists (Type I error).
What confidence level should I use?
95% confidence (p < 0.05) is the industry standard for most web A/B tests and is appropriate when the cost of a wrong decision is moderate. Use 99% confidence (p < 0.01) for high-stakes decisions — pricing changes, checkout flow changes, major UI overhauls — where a false positive is expensive. 90% confidence is sometimes used for exploratory tests where speed matters more than certainty, but it carries a 1-in-10 false positive rate.
How large does my sample size need to be?
The required sample size depends on three inputs: your baseline conversion rate, the minimum lift you want to detect (minimum detectable effect, or MDE), and your desired confidence and power levels. The formula is approximately n = (Z_α/2 + Z_β)² × 2p(1−p) / (MDE)². For a 95% confidence, 80% power test detecting a 10% relative lift on a 5% baseline conversion rate, you need roughly 15,000 visitors per variant. Use a sample size calculator before starting your test.
What is a p-value?
The p-value is the probability of observing a difference at least as large as the one you measured, assuming the null hypothesis is true — i.e., assuming both variants perform identically. A p-value of 0.03 means: if A and B were actually equal, you would see a difference this large or larger only 3% of the time by chance. It does not mean there is a 97% probability that B is better — a common misinterpretation.
Can I stop an A/B test early if I see a winner?
No — this is called the 'peeking problem' and it dramatically inflates your false positive rate. Statistical values fluctuate naturally during a test, especially early on. If you check significance repeatedly and stop as soon as p < 0.05, the true false positive rate can exceed 50% by the time you reach your planned sample size. Run your test to the predetermined sample size (or time period) before evaluating results.
What is the minimum detectable effect (MDE)?
The MDE is the smallest improvement you care enough to detect reliably. Choosing a smaller MDE requires a proportionally larger sample size. If your baseline conversion rate is 5% and your MDE is a 10% relative improvement (to 5.5%), you need far more data than if your MDE is a 50% relative improvement (to 7.5%). Set your MDE based on what lift would actually change your decision — not the smallest theoretically possible improvement.
Is my test data kept private on this site?
Yes. The A/B Test Calculator is 100% browser-based. All statistical calculations — Z-score, p-value, confidence intervals, sample size requirements — run locally in JavaScript on your device. None of your conversion counts, visitor numbers, or test details are ever sent to any server.