CRO A/B Testing Win Rates and What They Really Mean

Earlier this week, I mentioned how low A/B testing win rates can be.

After digging deeper, I realised that it’s even lower than the 20-30% range I said.

When you narrow it down to statistically significant winners tied to primary commercial metrics, the benchmark sits at just 10%.

This piqued our curiosity about our own win rate.

So we pulled our A/B testing results across all clients over the past year.

The result?

58.86% of our A/B tests were statistically significant winners.

And to be fair, that is a strong number.

But the number itself is only part of the story.

Because if one agency says its win rate is X

And another says theirs is Y…

That still does not tell a client which agency is better.

It only tells you the surface.

The real substance sits underneath it.

For me, there are 3 deeper layers to understand the win rate.

No. 1: Testing Necessity

• Was the A/B test necessary, or was this a sure bet?

• Or was it something that was bound to win because there was little to no downside risk, or a ton of industry proof that it works?

No. 2: Commercial Impact

• Did the winner materially move a growth metric?

• Did it create enough upside and ROI to justify the cost of building and testing it?

No. 3: Lessons Learnt

• What did we learn from the winners, and maybe even more importantly, what did we learn from the losers?

• How are those lessons shaping the roadmap, changing our priorities, and improving the next round of tests?

Honestly, that is why I’m comfortable talking about our own win rate.

Because it reflects the work we put into our CRO.

And we have substance beneath it.

• We are meticulous about our Insights.

• We are meticulous about our Recommendations.

• We run rigorous prioritisation using the PECTI Prioritisation Framework.

• We are constantly running learning loops from tests across our client base.

All of that helps give us a clearer understanding of what is actually worth testing.

So yes, win rate matters.

But only when you understand what sits underneath it.

Chat soon,

Peter