FREE A/B TEST CALCULATOR
Check statistical significance without confusing it with business impact
Compare conversion rates for a control and variant using a two-sided two-proportion z-test. See conversion rates, absolute difference, relative lift, z-score and p-value.
Experiment data
TEST RESULT
Enter both variants to analyze the observed difference.
METHOD
What this A/B test significance calculator does
The calculator runs a pooled two-proportion z-test. Under the null hypothesis, it assumes the two variants have the same underlying conversion rate and asks how surprising the observed difference would be if that assumption were true.
z = (Rate B − Rate A) ÷ pooled standard error
HOW TO USE IT
Enter comparable, independent samples
Define one conversion
Use the same success event for both variants: click, signup, purchase, or another binary outcome.
Choose the threshold first
Set 90%, 95% or 99% before interpreting the data rather than changing the bar after seeing results.
Read effect and evidence
Review absolute difference and relative lift alongside the p-value. Statistical significance does not tell you whether the effect is commercially meaningful.
INTERPRETATION
Statistical significance is not the same as practical significance
A very large experiment can make a tiny effect statistically significant. A small experiment can fail to detect a useful effect. Before shipping a change, ask whether the effect is large enough to matter, whether the test ran through relevant business cycles, whether allocation was randomized, and whether you defined the stopping rule before repeatedly checking the result.
For email experiments, automated opens make open-rate tests particularly difficult to interpret. Clicks, conversions or revenue may provide stronger downstream outcomes depending on the hypothesis.
COMMON ERRORS
Why A/B tests produce false confidence
Stopping when p crosses .05
Repeatedly peeking and stopping at the first favorable threshold can inflate false-positive risk.
Testing many variants
Multiple comparisons increase the chance of finding a seemingly significant result by chance and may require adjustment.
Ignoring business value
A conversion lift can coexist with lower order value or profit. The statistical test only evaluates the outcome you entered.
WORKED EXAMPLE
A/B significance calculation example
If control A has 10,000 recipients and 400 conversions while variant B has 10,000 and 520, the observed rates are 4.0% and 5.2%. The absolute difference is 1.2 percentage points and relative lift is 30%. These inputs cross a conventional 95% significance threshold in the two-sided test.
That is evidence against the equal-rate null hypothesis under the model assumptions. It does not prove the variant will always produce the same lift or that the lift is economically valuable.
CORE CONCEPTS
Confidence level, p-value and effect size
Threshold
A decision rule selected before interpretation. A 95% setting conventionally corresponds to alpha = 0.05 for this test.
p-value
Describes compatibility of the observed result with the no-difference model. It is not the probability that B is the true winner.
Effect size
Shows how large the observed change is. Review absolute difference and relative lift alongside significance.
EXPERIMENT DESIGN
What the calculator cannot repair after a bad experiment
Statistical arithmetic cannot fix biased allocation, changing eligibility, broken tracking, overlapping tests or selective stopping. Define the hypothesis, primary metric, eligible population, allocation and stopping rule before launch.
Sample size and power
An inconclusive result can reflect a small true effect, no meaningful effect, or insufficient sample size. Prospective sample-size planning also requires a baseline rate, minimum detectable effect, significance threshold and desired power. This calculator analyzes observed counts; it does not replace power analysis.
EMAIL-SPECIFIC QA
Before trusting an email A/B test
- Randomly allocate comparable eligible recipients.
- Change the variable tied to the hypothesis.
- Use comparable delivery windows.
- Prefer downstream outcomes when open tracking is unreliable.
- Check for automated click activity.
- Do not redefine the primary outcome after seeing results.
- Review business value as well as statistical evidence.
FAQ
A/B testing questions
What p-value is statistically significant?
At a preselected 95% confidence threshold, the conventional cutoff is p ≤ 0.05. The threshold is a convention, not proof of business importance.
What does “not significant” mean?
It means the observed data did not provide enough evidence to reject the no-difference hypothesis at your selected threshold. It does not prove the variants are identical.
What is relative lift?
Relative lift expresses the change relative to the control rate. Moving from 4% to 5% is +1 percentage point but +25% relative lift.
Can I use this for email subject-line tests?
You can compare binary outcomes, but open-rate experiments have tracking limitations. Define the metric and test plan before launch.
Related resources
Email Subject Line Testing · CTR & CTOR Calculator · Email Marketing Metrics