A/B Test Significance Calculator

95% confidence is the common bar, a z-score of about 1.96, and below it you treat the result as inconclusive, not as a small win. Two things corrupt the read. Small samples produce unstable rates, so a test with a few hundred visitors per variant will rarely clear 95% even when the difference is real. And checking results repeatedly, then stopping the moment significance appears, inflates false positives. Decide the sample size up front and read the number once.

When experiment results live in a Ferra table, visitors and conversions per variant, appended as each test ends, you can ask in plain English which tests cleared 95% and keep a running log of what actually shipped.

Four counts go in: visitors and conversions for A, visitors and conversions for B. Out come each variant's conversion rate, the relative uplift of B over A, the z-score for the difference, and the confidence level that the gap is real rather than random noise.

Variant A: 10,000 visitors, 500 conversions. Variant B: 10,000 visitors, 600. Rate A is 500 / 10,000 = 5.0% and rate B is 600 / 10,000 = 6.0%, a relative uplift of (6.0 − 5.0) / 5.0 = 20%. The pooled rate treats both samples as one: (500 + 600) / (10,000 + 10,000) = 1,100 / 20,000 = 5.5%.

The standard error of the difference is √(0.055 × 0.945 × (1/10,000 + 1/10,000)) = √0.000010395 ≈ 0.00322. Divide the observed gap by that error for the z-score: 0.01 / 0.00322 ≈ 3.10. A two-tailed normal lookup for z = 3.10 gives roughly 99.8% confidence, a gap this large would almost never appear by chance between two identical variants.

Mismatched denominators wreck more tests than bad math does. Count unique visitors for one variant but sessions or pageviews for the other, or pull the two variants' numbers from different date ranges, and one rate gets artificially inflated or deflated. The z-score then faithfully reports a difference your instrumentation created, not your design change.

95% is where most teams declare a winner. Some accept 90% for low-risk changes like copy tweaks; some hold out for 99% when the change is expensive to reverse: pricing, say, or checkout flow. In z-score terms, about 1.96 corresponds to 95% and about 2.58 to 99%.

High confidence means "the difference is probably real," not "the uplift is exactly this size": the true effect could be smaller or larger than the measured one. And a low confidence level, say 70%, doesn't mean the variants are equal. It means you can't yet tell the difference from noise, which usually calls for more traffic rather than a coin-flip decision.

Significance rises with bigger effects and bigger samples; cleaner measurement helps just as much. Every lever below pulls one of those three.