The 95% Confidence Misconception
When your A/B testing tool says a result is "95% statistically significant," most people interpret this as: "There's a 95% chance the winning variant is actually better." That interpretation is wrong, and acting on it leads to bad decisions.
What 95% statistical significance actually means: if the null hypothesis is true (meaning there is no real difference between variants), there is only a 5% probability of observing a result as extreme as the one you got. It's a statement about the data given the hypothesis, not the hypothesis given the data.
This distinction might sound academic, but it has real consequences. It means that even with proper testing methodology, 1 in 20 tests will show a "significant" result that's actually noise. And most teams run far more than 20 tests, meaning false positives are inevitable.
The Four Mistakes That Invalidate Results
1. Peeking at Results Too Early
This is the most common and most damaging mistake. You launch a test on Monday, check results on Wednesday, see that variant B is "winning" with 93% confidence, decide to check again Thursday, and now it's at 96% confidence. You call the test.
The problem: every time you check results and consider stopping, you're effectively running a new statistical test. This is called the multiple comparisons problem, or more colloquially, "peeking." Research from Optimizely's own statistics team demonstrated that checking results after every few hundred visitors can inflate false positive rates from the intended 5% to as high as 30%.
The fix: Determine your required sample size before starting the test and don't make any decisions until you reach it. If you need to monitor results early (for safety or technical reasons), use sequential testing methods or always-valid p-values that account for continuous monitoring.
2. Ending Tests Too Early
Related to peeking, but with a different cause. You see a big early effect, variant B is converting at 20% higher than control with 97% confidence after just 500 visitors. You stop the test to capitalize on the improvement.
Early results are inherently noisy. With small sample sizes, random variation can create large apparent effects that regress toward zero as more data accumulates. This phenomenon, called regression to the mean, means early "big winners" are often much smaller winners, or not winners at all, when measured over adequate sample sizes.
The fix: Calculate your minimum sample size using a power analysis before launching. For most A/B tests, you need to decide on three inputs: your baseline conversion rate, the minimum detectable effect (the smallest improvement worth caring about), and your desired statistical power (typically 80%). Online calculators like Evan Miller's sample size calculator make this straightforward.
3. Testing Too Many Variants
When you test five variants against a control, you're running five simultaneous comparisons. The probability that at least one shows a false positive isn't 5%, it's approximately 23% (calculated as 1 minus 0.95 to the fifth power). The more variants you test, the more likely random noise produces a "winner."
The fix: Apply a correction for multiple comparisons. The Bonferroni correction divides your significance threshold by the number of comparisons (so 0.05/5 = 0.01 per comparison). This is conservative but safe. Alternatively, limit tests to two or three variants total, which keeps the mathematics simpler and reduces the risk of false positives.
4. Ignoring Segments and Hunting for Significance
Your test didn't reach significance overall, so you start slicing the data: maybe it's significant for mobile users, or for users from paid traffic, or for users in California. If you check enough segments, you'll find one where the result is significant, by pure chance.
The fix: Pre-register your segments of interest before running the test. If you want to analyze results by device type, decide that before launching and account for it in your sample size calculation. Post-hoc segment analysis is fine for generating hypotheses, but never for drawing conclusions. Test those hypotheses in a follow-up experiment.
Practical Sample Size Guidelines
To detect a 5% relative improvement in a 3% baseline conversion rate (from 3.0% to 3.15%) with 80% power and 95% confidence, you need approximately 270,000 visitors per variant. That's over half a million total visitors for a simple A/B test.
For many sites, this means:
- Tests need to run for weeks, not days
- You need to focus on larger effects (10-20% relative improvements) if you want faster results
- Testing should target high-traffic pages where you can accumulate sample sizes quickly
- Minor tweaks (button color changes) rarely produce detectable effects unless your traffic is very high
This reality check disappoints teams who want to test every idea quickly, but it protects you from acting on noise and implementing changes that don't actually help.
Beyond Statistical Significance: Practical Significance
A result can be statistically significant but practically meaningless. With enough traffic, even a 0.01% improvement in conversion rate will achieve statistical significance. But if implementing the change requires engineering effort and the improvement adds $50 per month in revenue, it's not worth it.
Always pair statistical significance with practical significance: is the effect large enough to matter for your business? Define your minimum detectable effect before running the test, this is the smallest improvement that would justify the effort of implementing the change.
Bayesian vs. Frequentist Approaches
Traditional A/B testing uses frequentist statistics (p-values, confidence intervals). Many modern testing platforms, including some features in Spectry, offer Bayesian analysis as an alternative.
Bayesian methods have practical advantages:
- They naturally handle continuous monitoring without the peeking problem
- They provide intuitive probability statements ("there is a 92% probability that variant B is better")
- They can incorporate prior knowledge about expected effect sizes
- They make it easier to decide when to stop a losing test without guilt
The tradeoff: Bayesian methods require choosing priors, and they can be more computationally expensive. But for most practical A/B testing scenarios, they produce more actionable results than frequentist methods.
A Checklist for Trustworthy A/B Tests
Before launching any test, answer these questions:
- What is the baseline conversion rate for the metric you're testing?
- What is the minimum effect size you care about?
- How many visitors per variant do you need (from power analysis)?
- How long will it take to reach that sample size?
- Does the test duration cover at least one full business cycle (typically one week)?
- Have you pre-registered any segment analyses?
- Have you decided in advance what result will trigger implementation?
Spectry's A/B testing feature includes built-in sample size calculators and significance monitoring to help you avoid the most common pitfalls. But no tool can prevent bad methodology. That requires discipline and understanding.
The goal of A/B testing isn't to find "winners." It's to make better decisions. Sometimes the best decision is recognizing that you don't have enough data to decide, and waiting until you do.