# A/B Test Significance Calculator

> Tests whether a variant's conversion rate differs from the control's using the pooled two-proportion z-test, and reports both rates, absolute and relative lift, z-score, two-sided p-value, a confidence interval for the difference and the sample size needed per variant to detect the observed lift with 80 % power.

- Calculator id: `ab-test-significance` · Category: Business & Marketing (`business`) · Tool name: `calculate_ab_test_significance`
- Canonical page: https://tttkmbb.com/business/ab-test-significance · This document: https://tttkmbb.com/business/ab-test-significance.md · JSON definition: https://tttkmbb.com/business/ab-test-significance.json

## Purpose

Tests whether a variant's conversion rate differs from the control's using the pooled two-proportion z-test, and reports both rates, absolute and relative lift, z-score, two-sided p-value, a confidence interval for the difference and the sample size needed per variant to detect the observed lift with 80 % power.

**Use when:** You have visitors and conversions for a control and a variant (A/B test, split test, email test) and need to know whether the difference is statistically significant.

**Do not use when:** The metric is a mean (revenue per user, time on page) rather than a conversion proportion, or there are more than two variants (use a chi-square or multiple-comparison procedure).

## Input

| Parameter | Type | Unit | Required | Description |
| --- | --- | --- | --- | --- |
| `control_visitors` | integer |  | required | Number of users exposed to the control (A). (> 0, max 1000000000000) |
| `control_conversions` | integer |  | required | Conversions in the control group. (min 0, max 1000000000000) |
| `variant_visitors` | integer |  | required | Number of users exposed to the variant (B). (> 0, max 1000000000000) |
| `variant_conversions` | integer |  | required | Conversions in the variant group. (min 0, max 1000000000000) |
| `confidence_level_percent` | number | % | optional, default 95 | Confidence level for the significance decision and interval (95 → α = 0.05, two-sided). (> 0) |

## Output

| Field | Type | Unit | Description |
| --- | --- | --- | --- |
| `control_rate_percent` | number | % | control_conversions / control_visitors × 100. |
| `variant_rate_percent` | number | % | variant_conversions / variant_visitors × 100. |
| `absolute_lift_percentage_points` | number | pp | variant rate − control rate, in percentage points. |
| `relative_lift_percent` | number | % | (variant rate − control rate) / control rate × 100. |
| `standard_error_percentage_points` | number | pp | √(p̄(1 − p̄)(1/n1 + 1/n2)) × 100 under the null hypothesis. |
| `z_score` | number |  | (p2 − p1) / pooled standard error; positive when the variant is higher. |
| `p_value` | number |  | Probability of a difference at least this large if both rates were equal. |
| `significant` | boolean |  | true when p_value < 1 − confidence_level/100. |
| `ci_lower_percentage_points` | number | pp | Lower bound of the confidence interval for variant − control (unpooled Wald interval). |
| `ci_upper_percentage_points` | number | pp | Upper bound of the same interval. |
| `required_visitors_per_variant` | integer |  | Visitors per group needed to detect the observed difference with 80 % power at this confidence level (rounded up). |
| `verdict` | string |  | Plain-language conclusion. |

## Formula

`p1 = x1/n1, p2 = x2/n2, p̄ = (x1 + x2)/(n1 + n2); z = (p2 − p1) / √(p̄(1 − p̄)(1/n1 + 1/n2)); p_value = 2·(1 − Φ(|z|)); CI = (p2 − p1) ± z_crit·√(p1(1 − p1)/n1 + p2(1 − p2)/n2); n per variant = (z_crit·√(2·p̄(1 − p̄)) + z_0.80·√(p1(1 − p1) + p2(1 − p2)))² / (p2 − p1)²`

Pooled two-proportion z-test (normal approximation; adequate when each group has at least about 10 conversions and 10 non-conversions). The sample-size formula is the standard two-sample proportion formula (Fleiss) without continuity correction, with p̄ the mean of the two rates.

## Data Sources

- NIST/SEMATECH e-Handbook of Statistical Methods, 7.3.3 Do two processes produce the same proportion of defectives? — https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm (government, retrieved 2026-09-24)
- Wikipedia – Two-proportion Z-test — https://en.wikipedia.org/wiki/Two-proportion_Z-test (reference, retrieved 2026-09-24)
- Kohavi R, Tang D, Xu Y (2020) Trustworthy Online Controlled Experiments, Cambridge University Press — https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59 (textbook, retrieved 2026-09-24)

Data freshness: `static`. Deterministic formula with fixed constants; results never go stale. Inputs supplied by the caller determine the output.

## API

- `GET https://tttkmbb.com/api/v1/calculate/ab-test-significance?control_visitors=…&control_conversions=…&variant_visitors=…&variant_conversions=…`
- `POST https://tttkmbb.com/api/v1/calculate/ab-test-significance` with JSON body `{"inputs": {…}}`
- Response: unified envelope (`success`, `request`, `result.values`, `result.units`, `sources`, `freshness`, `timestamp`, `next_actions`, `links`); see https://tttkmbb.com/docs/response-format.md
- Schema: https://tttkmbb.com/api/v1/calculators/ab-test-significance · OpenAPI operationId `calculate_ab_test_significance` in https://tttkmbb.com/openapi.json
- Authentication: none. Rate limit: fair use, see https://tttkmbb.com/docs/rate-limits.md.

## MCP

- Server: `https://tttkmbb.com/mcp` (Streamable HTTP, JSON-RPC 2.0, no auth)
- Tool: `calculate_ab_test_significance` (dedicated) or `run_calculator` with `{"calculator_id": "ab-test-significance", "inputs": {…}}`

## Example

- 1,000/50 vs 1,000/70 at 95 %: inputs `{"control_visitors":1000,"control_conversions":50,"variant_visitors":1000,"variant_conversions":70}` → `{"control_rate_percent":5,"variant_rate_percent":7,"absolute_lift_percentage_points":2,"relative_lift_percent":40,"standard_error_percentage_points":1.0621,"z_score":1.8831,"p_value":0.0597,"significant":false,"ci_lower_percentage_points":-0.08,"ci_upper_percentage_points":4.08,"required_visitors_per_variant":2213}`
- 5,000/200 vs 5,000/260 at 95 %: inputs `{"control_visitors":5000,"control_conversions":200,"variant_visitors":5000,"variant_conversions":260}` → `{"control_rate_percent":4,"variant_rate_percent":5.2,"relative_lift_percent":30,"z_score":2.8642,"p_value":0.0042,"significant":true,"ci_lower_percentage_points":0.38,"ci_upper_percentage_points":2.02,"required_visitors_per_variant":4783}`

```
GET https://tttkmbb.com/api/v1/calculate/ab-test-significance?control_visitors=1000&control_conversions=50&variant_visitors=1000&variant_conversions=70
```

## Limitations

The metric is a mean (revenue per user, time on page) rather than a conversion proportion, or there are more than two variants (use a chi-square or multiple-comparison procedure). Pooled two-proportion z-test (normal approximation; adequate when each group has at least about 10 conversions and 10 non-conversions). The sample-size formula is the standard two-sample proportion formula (Fleiss) without continuity correction, with p̄ the mean of the two rates. All values are computed from the formula above; no measurement or live data is involved.

## FAQ

**Why is a 40 % lift not significant?**

With 1,000 visitors per group the rates 5 % and 7 % have a pooled standard error of about 1.06 points, so the 2-point difference is only 1.88 standard errors away from zero (p = 0.06). About 2,213 visitors per group would be needed to detect it reliably.

**One-sided or two-sided?**

Two-sided: it tests for a difference in either direction. For a one-sided test (variant better only) halve the p-value.

**Does the confidence interval match the p-value?**

Approximately: the interval uses the unpooled standard error while the test uses the pooled one, so near the threshold they can disagree slightly.

## Related

- [Conversion Rate Calculator](https://tttkmbb.com/business/conversion-rate.md) — Conversion rate of a single variant.
- [Sample Size Calculator](https://tttkmbb.com/statistics/sample-size.md) — Plan a sample size for a target margin of error.
- [Normal Distribution Calculator](https://tttkmbb.com/statistics/normal-distribution.md) — Look up the normal probabilities behind the z-score.