A Quick Guide to A/B Testing: What to Expect from Your CRO Agency

What should you expect from A/B testing agency support? In short: transparency, realism and rigour.

A/B testing is a blend of art and science, skewing heavily towards the latter. It is as much about finding answers and drawing conclusions as it is simply chasing wins.

We share this quick guide to A/B testing with every Blend Commerce client to help explain our approach and how that translates to getting the best value from A/B testing. As many clients have found it useful in improving their understanding of A/B testing and building a more effective experimentation culture, we decided to publish it.

Why Do A/B Testing?

A/B testing helps us make decisions using real customer behaviour rather than assumptions. It is normal for many experiments not to produce a winning variation.

In fact, industry data suggests that only around 10–20% of tests typically achieve a statistically significant improvement. However, every conclusive result gives us useful insight into what customers need and how we should shape future optimisation work.

From January 2025 to March 2026, Blend ran 159 A/B tests and achieved a 58.86% win rate. While we are proud of this performance, we will never expect every test to win. A losing or inconclusive result still protects the business from implementing an unproven change and gives our strategists evidence to refine the next hypothesis.

Why We Wait for Statistical Significance

Early test results can move considerably as more visitors enter the experiment, so a variation that appears to be losing may later recover or win. Statistical significance (i.e. the point at which you can trust the data you're seeing) helps us judge whether the difference is likely to reflect genuine customer behaviour rather than normal fluctuations in the data.

Why You Shouldn't End a Test Early Just Because It's Not Winning

Tests commonly run for around two weeks, but the exact time depends on traffic, conversion volume and the size of the difference between the variations.

We therefore recommend allowing each experiment to run for the planned period and getting as close as possible to statistical significance before making a decision, unless there is a clear commercial or customer-experience reason to stop it early.

Losing Tests Are Not Failures

Optimising anything is a process of elimination. It would be great if we just knew everything that's going to work for your store, but the reality is that optimising an ecommerce website is about sorting the changes that make positive impacts from the ones that have negative or no impact at all.

That's what your CRO agency should be doing with A/B testing - validating ideas. Of course, aiming to win with as many as possible, but also taking valuable lessons from losing tests. Gaining certainty about what won't work for your store can be as valuable as knowing what will.

Further Reading

Why most A/B tests fail, and why that is OK – Oliver Palmer
Experimentation success through failed tests – Optimizely
How long to run an experiment – Optimizely

More about A/B Testing from the Blend Blog

What Is a Marketing Experiment?

15 A/B Test Ideas to Improve eCommerce Conversions

A/B Testing vs Multivariate Testing: When to Use Each for Shopify CRO