Statistics & calling a winner

Spectry uses a Bayesian model rather than classic p-values. Instead of "is this difference statistically significant?", it answers a more directly useful question: what is the probability that this variation is the best one?

The columns

Column	What it means
Visitors	Unique visitors who saw this variation.
Conversions	How many of them converted on your goal.
Conv. Rate	Conversions ÷ visitors, as a percentage.
95% CI	The credible interval — the range the true conversion rate is 95% likely to fall in. A wide range means you don't have enough data yet.
Prob. Best	The probability this variation beats every other one. The bar turns green at 95%.
Lift vs Control	Relative change in conversion rate against control.
Expected Loss	How much conversion rate you'd give up, on average, if you shipped this variation and it turned out not to be the best. Small expected loss is the practical signal that a choice is safe.

When a winner is declared

The WINNER badge appears only when both are true:

• every variation has at least 100 visitors, and
• one variation's Prob. Best is 95% or higher.

Until the first condition is met the banner reads "Need at least 100 visitors per variation" and lists where each one stands. Once there's enough data but no variation is clearly ahead, it reads "No clear winner yet. Continue running the test." — which is a real result, not a failure.

100 visitors per variation is the floor, not the target. It's the minimum before Spectry will name a winner at all. A test on a 2% conversion rate needs thousands of visitors per variation before the intervals separate.

Sample size calculator

Before launching, use the Sample Size Calculator on the test to find out whether the experiment is worth running:

• Baseline conv. rate (%) — what the control does today.
• Min. detectable effect (%) — the smallest relative improvement you'd care about. Asking to detect 2% costs enormously more traffic than 10%.
• Statistical power (%) — how often you want to catch a real effect (80% is the usual choice).
• Number of variations — including control.
• Daily visitors — used to turn the sample size into an Estimated duration.

It returns Per variation and Total needed. If the estimated duration is three months, either raise the minimum detectable effect, test something bolder, or test a page with more traffic.

Reading it honestly

• Don't peek and stop. Checking daily is fine; stopping the moment a variation crosses 95% is how you ship noise. Decide the run length up front and let it finish.
• A 60% Prob. Best is not a lead. With two variations, 50% is a coin flip; 60% is barely distinguishable from one.
• Overlapping credible intervals mean "not yet". If control is 3.1–4.2% and the variation is 3.4–4.6%, you don't have a result regardless of what the Conv. Rate column says.
• More variations, more traffic. Each extra variation splits your traffic further and raises the bar for any of them to stand out.