Interpreting A/B Test Results: A Guide to Statistical Significance

Statistical significance in A/B testing is one of the thorniest parts of conversion rate optimisation, and strikes fear into the heart of many a season ecommerce pro.

Here’s a scenario for you: a Shopify product page test is looking positive after four days. The testing platform shows that the variant’s conversion rate is 14% ahead. The dashboard says the result has reached 95% confidence, so someone quite reasonably asks whether the change can go live.

What do you reckon?

Well, the test might be reliable. But it might also be too early, for a number of reasons: a few extra orders may have landed in one version by chance, a campaign may have changed the traffic mix halfway through the test, or one version might have skewed towards mobile users (albeit that should be controlled by your A/B testing platform).

So yes, right now it looks like that variation is better, but it’s not full time; the game is still being played. This one might win, but the original might make a comeback, or you might need to go into extra time.

This guide explains how to read A/B test results without needing a statistics background. It covers what statistical significance means, what it does not mean, which numbers to check, and how to decide whether a test result is strong enough to use.

If you are setting up your first test, read our guide on how to run your first A/B test on Shopify. If you already have a result and need to decide what to do next, start here.

What Statistical Significance Means in A/B Testing

Statistical significance in an A/B test means the difference between the variant (B) and the control (A) is large enough, compared with the amount of data collected, that you can be confident in the result, with random variation no longer a convincing explanation. It means you can trust what you’re seeing.

It works like this:

In an A/B test, you show one version of a page to one group of visitors and another version to another group.

The control is the existing version. The variant is the changed version. The test measures whether one version performs better on a chosen metric, such as purchases, add-to-cart rate, checkout starts, or revenue per visitor.

Statistical significance helps you judge whether the difference between the two versions is likely to be real, rather than ordinary random variation.

That is important because split testing is still uneven. One hour can produce more orders than the next. One day can bring more ready-to-buy visitors than another. Even if two page versions are equally good, one version can appear to be ahead for a while simply because of normal variation in traffic and buying behaviour.

When Do You Reach Statistical Significance?

A common threshold for statistical significance is 95% confidence, often shown as a p-value below 0.05.

A p-value of 0.05 means this:

If there were no real difference between the control and the variant, a result this large would happen about five times in 100 similar tests just because of random variation.

That does not mean there is a 95% chance the variant is better. It also does not mean the result is worth implementing.

The American Statistical Association has even warned against this misunderstanding. A p-value does not tell you whether a hypothesis is true. Statistical significance also does not tell you whether the result is “significant” enough to be rolled out.

For CRO, that second point is where many poor decisions happen. A test can be statistically significant and still be commercially wrong.

Why 95% Confidence Is Not Enough on Its Own

A/B testing dashboards often use winner labels, confidence percentages, coloured charts, and uplift figures. These look pleasing and they can be handy, but they can also make a result look more settled than it is.

Before rolling out a variant, you need to check the result from several angles.

The result needs these:

• enough traffic

• enough conversions

• the test to have run for long enough to cover normal buying behaviour (i.e. ruling out a sale-related or seasonal anomaly)

• the tracking to be reliable

• the uplift to be large enough to justify the work for rollout

Beware isolated results in isolated metrics too; a variant can win on conversion rate but also lower your average order value, just the same as a product page message can work well for returning customers and do very little for first-time visitors.

This is why the primary metric is very, very important.

If the test was designed to improve purchases, then judge it mainly on purchases. If the test was designed to improve revenue per visitor, don’t get distracted by great results in some click metric that’s not directly related. Supporting metrics are useful, but don’t let them take over.

Our guide to A/B testing benchmarks for ecommerce explains why different metrics need different expectations. A click uplift is easier to get than a purchase uplift. A revenue-per-visitor uplift is usually harder again.

Subscribe to the Shopify CRO newsletter

How to Read A/B Test Results Before Rolling Out

Read the result in this order:

First, check how many visitors or sessions saw each version. A test with 300 visitors per version is usually too fragile for a purchase-based decision. A few orders can change the result heavily.

Next, check the absolute number of conversions - don’t just look at the conversion rate or uplift percentage. A 75% increase in conversion rate sounds great, but it can take on a different complexion if you know it’s come from four conversions in the control and seven in the variant.

Then look at the absolute difference. If the control converted at 2.0% and the variant converted at 2.2%, the absolute increase is 0.2 percentage points. The relative uplift is 10%. The relative number sounds larger, but the absolute number gives you a better feel for what changed.

After that, check the confidence interval if your testing tool provides one. A confidence interval shows the range of results that are still plausible based on the data. If the observed uplift is 12%, but the range includes a small loss, the result is not as settled as the headline number suggests.

Finally, compare the result with the minimum worthwhile lift.

Minimum worthwhile lift means the smallest improvement you would actually care about. A 1% relative uplift may be worth rolling out on a checkout step with large traffic and low implementation risk. The same uplift may not be worth a complicated theme change on a lower-traffic product template.

This is separate from the minimum detectable effect, often called MDE.

Minimum detectable effect is the smallest change your test is designed to detect. Smaller effects need more traffic. If your store does not have enough traffic, the test may only be able to detect a large improvement within the reasonable testing timeframe.

That is why many low-traffic Shopify tests end with an inconclusive result. The test might be based on a brilliant idea, but the store may just not have enough data to measure a small change reliably. This is also why we only carry out A/B testing on stores with at least 50,000 sessions per month.

Why Sample Size Is Important

A/B testing works better when the test has enough data to separate real performance from normal variation.

Imagine two versions of a product page:

The control receives 500 visitors and gets 10 orders. The variant receives 500 visitors and gets 14 orders. The variant looks 40% better.

That sounds good, but you can also see that in absolute terms it’s only four extra orders. You can’t make business decisions based on four orders.

Those four orders could be caused by the page change, but they could also be caused by traffic quality, returning customers, device mix, discount behaviour, or plain old random chance.

Now imagine the same conversion rates across 50,000 visitors per version. The result becomes more persuasive because it’s based on a much larger sample. One or two unusual orders no longer move the result as much - you’re talking about an absolute number of 400, which is clearly a much safer interval.

This is why an A/B test sample size calculator is useful before the test starts. It helps you estimate how much traffic you need based on your current conversion rate and the size of uplift you want to detect, helping you to plan.

Why Peeking Creates False Winners

Peeking means checking a test repeatedly and stopping as soon as the result looks significant. Just. Don’t. Do it.

It’s one of the most common ways A/B test results become unreliable.

By all means, have a look to check that the test is working. You should still monitor live tests for broken layouts, tracking errors, stock issues, or obvious performance drops or spikes (we recently caught a strange drop in a client’s A/B test and found that they had switched off all paid traffic one week in without us knowing - you want to catch this stuff).

The problem arises when you change the stopping point because the dashboard has turned green.

Many standard significance calculations assume the sample size was decided before the test started. If the real rule is “keep checking until the variant wins”, the test has become easier to misread and it’s not a fair test at all.

Evan Miller’s article, How Not to Run an A/B Test, explains this well. Repeatedly checking significance increases the chance of finding a false winner. A/B testing isn’t quite a science, but you should treat as much like science as possible - that means you are looking for truth; not to be proven right. More on this in our article on how to build a culture of CRO experimentation in your business tk /blogs/shopify/building-culture-experimentation-ecommerce.

For ecommerce stores, runtime and sample size are important. A test should usually cover a normal trading cycle. Weekdays and weekends can behave differently. Email campaign days can behave differently. Payday, stock levels, paid media changes, and discounting can all affect buying behaviour.

A test that starts on Tuesday and stops on Friday may have enough sessions, but it may not have seen enough of the store’s normal rhythm.

What Sample Ratio Mismatch Means

Sample ratio mismatch means the test did not split traffic the way it was meant to.

If a test is set to split traffic 50/50, the two versions should receive roughly equal traffic. They will rarely be perfectly equal, but a large difference is a warning sign.

For example, 7,700 to one and 8,000 to another is ok, but if the control receives 8,000 visitors and the variant receives 5,500 visitors in a 50/50 test, something may be wrong.

Possible causes include targeting settings, cookie behaviour, redirects, consent settings, app conflicts, or analytics filtering.

Sample ratio mismatch is important because the test may no longer be comparing two fair groups. You can still calculate conversion rates, but the result becomes harder to trust.

If you see a large traffic split problem, investigate before acting on the result. Often the cautious route is to fix the setup and rerun the test.

How to Decide Whether an A/B Test Result Is Worth Acting On

A test result is worth acting on when the evidence is strong enough and the change makes business sense.

A good result that has achieved statistical significance in an A/B test usually has these features:

• The test reached the planned sample size

• The test ran long enough to cover normal buying behaviour

• The traffic split looks correct

• The primary metric improved

• The result is large enough to have a commercial impact

• There were no major tracking issues, bugs, stock problems, or campaign problems during the test

If those checks are satisfied, rollout is reasonable. Monitor the change after rollout, because live site conditions can differ from test conditions.

If the result is statistically significant but the commercial gain is tiny, treat it carefully.

For example, a Shopify store with 100,000 monthly product page sessions, a 2.5% conversion rate, and a $70 average order value gets around 2,500 orders from those sessions.

A 1% relative uplift would add about 25 orders per month. That is around $1,750 in extra monthly revenue before returns, discounts, fees, fulfilment costs, and margin.

That may be worth taking if the change is quick and low-risk. It may not be worth taking if this change requires complex ongoing maintenance or additional human resources.

CRO decisions should account for effort as well as uplift. That is the thinking behind how we prioritise CRO recommendations.

What to Do With Inconclusive A/B Test Results

An inconclusive result means the test did not provide enough evidence for a confident decision.

There are two common reasons:

Reason 1: the change probably did very little. The result sits close to zero, and the confidence interval is narrow enough to suggest there is no meaningful movement.

Reason 2: the test was underpowered. The variant may look better, but the sample is too small to know whether the improvement is real.

These situations need different responses.

If the result suggests the change did very little, log the result and move on.

If the test was underpowered, look at the original reason for running it. If the idea came from strong customer evidence, such as repeated survey feedback, support tickets, or session recordings, it may deserve a stronger second version. If the idea was based on a small preference or internal opinion, it may not deserve more testing time.

A testing library helps here. Blend’s Shopify A/B test results library shows how different tests feed into a wider CRO programme, rather than being treated as isolated wins or losses.

What Low-Traffic Shopify Stores Should Do

We’re often asked this: what can I do if my traffic is less than 50,000 sessions a month? Well, low-traffic stores can run A/B tests, but they need to be highly selective. Above all, we’d recommend you use your resources to increase your traffic first, but we can also carry out a heuristic CRO audit.

If you really want to test, and you’re between 30,000 and 50,000 sessions, bear in mind that small changes on low-traffic pages are hard to measure. A button copy change on a product page with limited traffic may need months to produce a reliable purchase result. By then, the store may have changed pricing, campaigns, stock, or seasonality.

Higher-traffic areas are better candidates. Sitewide messaging, cart drawer changes, navigation changes, delivery information, collection page filters, and checkout-adjacent messaging may produce enough data more quickly.

The change also needs to be big enough to measure. A small design tweak is harder to detect than a clearer delivery message, a better product comparison section, or a more useful subscription explanation.

Use A/B testing where the answer is genuinely uncertain and the store has enough traffic to measure the result.

For inspiration, use our list of ecommerce A/B test ideas.

Subscribe to the Shopify CRO newsletter

Frequentist, Bayesian, and Sequential Testing

Most ecommerce teams don’t need to master the maths behind every testing method. But you do need to understand what your testing tool is claiming.

What is a Frequentist Test in A/B Testing?

A frequentist test usually reports a p-value or confidence level. It works best when the sample size, primary metric, and stopping point are planned before the test starts.

What is a BayesianTest in A/B Testing?

A Bayesian test usually reports probability-style results, such as the probability that one version is better than another. This can be easier to read, but it still depends on the quality of the data and the assumptions in the model.

What is a SequentialTest in A/B Testing?

Sequential testing is designed to allow planned checks while the test is running. This is different from checking a standard test every morning and stopping when the result looks good.

The method is important, but the basics matter more. Poor tracking, weak sample size, sample ratio mismatch, and unclear stopping rules can damage any type of test.

What a Useful A/B Test Calculator Should Tell You

Most A/B test calculators ask for visitors and conversions, then return a significance result. Helpful, but it doesn’t give you all the answers you need for your ecommerce context.

A good A/B test calculator should help you decide whether the result is reliable enough to act on.

It should ask for:

• Control visitors or sessions

• Control conversions

• Variant visitors or sessions

• Variant conversions

• Test start date and end date

• Primary metric

• Expected traffic per day

• Minimum worthwhile lift

• Notes on campaigns, tracking changes, QA issues, stock issues, or unusual traffic

It should return:

• Conversion rate by version

• Absolute uplift

• Relative uplift

• P-value or confidence level

• Confidence interval

• Sample ratio mismatch warning

• Estimated sample still needed

• Estimated days remaining

• A decision note written in plain English

Useful decision notes would look like this:

“Promising, but underpowered. The variant is ahead, but the test has not collected enough conversions to support rollout.”

Or:

“Do not act on this result yet. The traffic split is far enough from the planned allocation to suggest a sample ratio mismatch. Check the setup before rerunning the test.”

That kind of A/B testing calculator would be more useful for DTC teams than a tool that only gives a green winner label.

How Blend Reads A/B Test Results

When Blend reviews an A/B test result, we start with the test record.

We check what changed, who saw the test, when it ran, and which metric was chosen before launch. We look for anything that may have affected the result, such as campaigns, stock issues, tracking changes, theme updates, app changes, or QA problems.

Then we look at the numbers.

A strong result can move into rollout. A positive but underpowered result may need another test. A statistically significant result with a tiny commercial gain may be logged but left out of development. A result with broken tracking or a serious sample ratio mismatch should usually be treated as invalid.

This is why CRO should be run as a program, rather than a string of disconnected tests. The value comes from building evidence over time: what customers respond to, what causes friction, which parts of the store are worth changing, and which ideas are too weak to keep testing.

If you want help building that kind of testing programme, read more about our Shopify conversion rate optimisation work. If you are unsure whether your store is ready for regular testing, a Shopify CRO audit can help find the highest-value opportunities before you invest in experiments.