Confidence explained

Spectry shows a "confidence" figure in two places, and they mean fundamentally different things. Mixing them up leads people to launch weak tests and to stop good ones early — so it is worth two minutes.

1. Proposal confidence — an evidence score

On an opportunity, confidence is a 0–100 evidence score. It answers: how well supported is this suggestion? It is assembled from factors such as how much data backs the observation, how consistent the pattern is across segments, and how cleanly the target element was identified. You can expand the score to see the factor breakdown.

This is not a statistical claim. An evidence score of 85 does not mean "85% likely to win". It means the suggestion rests on solid observation. Whether the change actually works is exactly what the test is for.

2. Test confidence — a win probability

Once a test is running, confidence is a Bayesian probability that the variation beats the control. This one is a statistical claim: 95% means that given the data collected so far, there is a 95% probability the variation is genuinely better. It moves as data arrives and is the number to use when deciding whether a test has finished.

Side by side

	Proposal confidence	Test confidence
Where	Opportunities	A/B test results
Means	Strength of the evidence behind the idea	Probability the variation beats control
Scale	0–100 evidence score	0–100% probability
Changes when	New data changes the observation	Continuously, as the test collects traffic
Use it to	Prioritise what to test first	Decide when to call a winner

What this means in practice

• A high proposal confidence is a reason to test the idea sooner, never a reason to skip testing it.
• A high-evidence proposal can still lose its test. That is a legitimate outcome and genuinely useful: it disproves an assumption you had good reason to hold.
• Only test confidence should end a test. See Goals & reading results.