Home › Business & Marketing › A/B Test Significance Calculator
A/B Test Significance Calculator
Tests whether a variant's conversion rate differs from the control's using the pooled two-proportion z-test, and reports both rates, absolute and relative lift, z-score, two-sided p-value, a confidence interval for the difference and the sample size needed per variant to detect the observed lift with 80 % power.
When to use
You have visitors and conversions for a control and a variant (A/B test, split test, email test) and need to know whether the difference is statistically significant.
Do not use when: The metric is a mean (revenue per user, time on page) rather than a conversion proportion, or there are more than two variants (use a chi-square or multiple-comparison procedure).
Formula
p1 = x1/n1, p2 = x2/n2, p̄ = (x1 + x2)/(n1 + n2); z = (p2 − p1) / √(p̄(1 − p̄)(1/n1 + 1/n2)); p_value = 2·(1 − Φ(|z|)); CI = (p2 − p1) ± z_crit·√(p1(1 − p1)/n1 + p2(1 − p2)/n2); n per variant = (z_crit·√(2·p̄(1 − p̄)) + z_0.80·√(p1(1 − p1) + p2(1 − p2)))² / (p2 − p1)²
Pooled two-proportion z-test (normal approximation; adequate when each group has at least about 10 conversions and 10 non-conversions). The sample-size formula is the standard two-sample proportion formula (Fleiss) without continuity correction, with p̄ the mean of the two rates.
Inputs
| Parameter | Type | Unit | Required | Description |
|---|---|---|---|---|
control_visitors | integer | yes | Number of users exposed to the control (A). Range: > 0, ≤ 1000000000000 | |
control_conversions | integer | yes | Conversions in the control group. Range: ≥ 0, ≤ 1000000000000 | |
variant_visitors | integer | yes | Number of users exposed to the variant (B). Range: > 0, ≤ 1000000000000 | |
variant_conversions | integer | yes | Conversions in the variant group. Range: ≥ 0, ≤ 1000000000000 | |
confidence_level_percent | number | % | default 95 | Confidence level for the significance decision and interval (95 → α = 0.05, two-sided). Range: > 0 |
Outputs
| Output | Type | Unit | Description |
|---|---|---|---|
control_rate_percent | number | % | control_conversions / control_visitors × 100. |
variant_rate_percent | number | % | variant_conversions / variant_visitors × 100. |
absolute_lift_percentage_points | number | pp | variant rate − control rate, in percentage points. |
relative_lift_percent | number | % | (variant rate − control rate) / control rate × 100. |
standard_error_percentage_points | number | pp | √(p̄(1 − p̄)(1/n1 + 1/n2)) × 100 under the null hypothesis. |
z_score | number | (p2 − p1) / pooled standard error; positive when the variant is higher. | |
p_value | number | Probability of a difference at least this large if both rates were equal. | |
significant | boolean | true when p_value < 1 − confidence_level/100. | |
ci_lower_percentage_points | number | pp | Lower bound of the confidence interval for variant − control (unpooled Wald interval). |
ci_upper_percentage_points | number | pp | Upper bound of the same interval. |
required_visitors_per_variant | integer | Visitors per group needed to detect the observed difference with 80 % power at this confidence level (rounded up). | |
verdict | string | Plain-language conclusion. |
Example
1,000/50 vs 1,000/70 at 95 %: {"control_visitors":1000,"control_conversions":50,"variant_visitors":1000,"variant_conversions":70} → {"control_rate_percent":5,"variant_rate_percent":7,"absolute_lift_percentage_points":2,"relative_lift_percent":40,"standard_error_percentage_points":1.0621,"z_score":1.8831,"p_value":0.0597,"significant":false,"ci_lower_percentage_points":-0.08,"ci_upper_percentage_points":4.08,"required_visitors_per_variant":2213}
5,000/200 vs 5,000/260 at 95 %: {"control_visitors":5000,"control_conversions":200,"variant_visitors":5000,"variant_conversions":260} → {"control_rate_percent":4,"variant_rate_percent":5.2,"relative_lift_percent":30,"z_score":2.8642,"p_value":0.0042,"significant":true,"ci_lower_percentage_points":0.38,"ci_upper_percentage_points":2.02,"required_visitors_per_variant":4783}
GET https://tttkmbb.com/api/v1/calculate/ab-test-significance?control_visitors=1000&control_conversions=50&variant_visitors=1000&variant_conversions=70
Machine access
- API:
GET https://tttkmbb.com/api/v1/calculate/ab-test-significance(query parameters) orPOSTwith a JSON body{"inputs": {...}} - Schema: https://tttkmbb.com/api/v1/calculators/ab-test-significance · Markdown: https://tttkmbb.com/business/ab-test-significance.md · JSON definition: https://tttkmbb.com/business/ab-test-significance.json
- MCP: server
https://tttkmbb.com/mcp, toolcalculate_ab_test_significance - OpenAPI operationId:
calculate_ab_test_significance - Freshness:
static. Authentication: none. Rate limit: fair use (see rate limits).
Sources
FAQ
Why is a 40 % lift not significant?
With 1,000 visitors per group the rates 5 % and 7 % have a pooled standard error of about 1.06 points, so the 2-point difference is only 1.88 standard errors away from zero (p = 0.06). About 2,213 visitors per group would be needed to detect it reliably.
One-sided or two-sided?
Two-sided: it tests for a difference in either direction. For a one-sided test (variant better only) halve the p-value.
Does the confidence interval match the p-value?
Approximately: the interval uses the unpooled standard error while the test uses the pooled one, so near the threshold they can disagree slightly.
Related calculators
- Conversion Rate Calculator — Conversion rate of a single variant.
- Sample Size Calculator — Plan a sample size for a target margin of error.
- Normal Distribution Calculator — Look up the normal probabilities behind the z-score.