What Questions to Ask a CRO Agency?

Hiring a CRO agency can be a good decision when your Shopify store has traffic, but the site is not turning enough of that traffic into revenue.

It can also become an expensive frustration if the agency turns out to be a bad fit. You now the classic agency pitfalls: prioritising work, explaining results, understanding your brand, turning research into live improvements. We’ve worked with countless brands who’ve experienced frustrations with these and more.

And I have seen this from both sides.

At Blend, we often speak to eCommerce brands after they have already worked with another CRO agency. The details differ, but the problems tend to follow a similar pattern:

• A/B tests were run, but most came back inconclusive

• There was no obvious order of priority

• The team did not get meaningful results

• Communication was unclear or too infrequent

• The recommendations felt generic

• The agency did not properly understand the brand, the product, or the customer

When these things happen, the issue is usually the operating model behind the work.

Before hiring a CRO agency, you need to ask questions that reveal how the agency thinks, how it prioritises, how it handles inconclusive results, and how it works with your internal team.

This article covers the questions I would ask if I were hiring a CRO agency for an eCommerce brand.

The Questions to Ask a CRO Agency

The most helpful questions to ask a CRO agency are:

• Do you do only strategy, or do you do CRO implementation too?

• How do you prioritise what to work on first?

• What evidence supports your recommendations?

• What happens when an A/B test fails or comes back inconclusive?

• How do you decide whether to test, implement, research, or reject an idea?

• How do you gauge and ensure statistical significance?

• How will you understand our brand and customer?

• How will communication and reporting work?

• Who will actually do the work?

• Which metrics do you optimise beyond conversion rate?

• Can you show case studies that match our type of problem?

• What are you not promising?

Each question gives you a different view of the agency.

You are looking for a team that can explain its method clearly, show relevant experience, and be honest about what CRO can and cannot control.

Be Careful With Agencies That Guarantee a Specific Uplift

Beware any guarantee on results. Agencies can and should guarantee you outputs, but they can never guarantee outcomes. If they do, chances are they’re just trying to tell you what they think you want to hear.

A CRO agency that says “we guarantee a 20% increase in conversion rate” might sound great, but the truth is a CRO agency does not control every factor that affects conversion performance.

Your results can be influenced by:

• Traffic quality

• Paid media performance

• Product demand

• Pricing

• Discounting

• Stock availability

• Seasonality

• Shipping costs

• Competitor activity

• Brand awareness

• Product margin

• Technical constraints

• Approval speed inside your business

An agency can control the quality of its research, its prioritisation, its implementation standards, its testing method, and its reporting.

No agency can control the 1,001 other commercial factors around your store.

A good, honest and realistic answer from a CRO agency would sound more like this:

“We cannot guarantee a fixed percentage uplift. We can show you how we diagnose issues, prioritise work, test with rigour, implement, and report on commercial impact. Based on a reliable benchmark for your specific sector and our typical results for clients that look like you, we will work to an expected uplift of X.”

Yes, that answer is a lot less instantly gratifying, but it is much more credible.

How Do You Prioritise What to Work on First?

This should be one of the first questions in your decision process.

Most eCommerce teams already have a long list of things they could improve. Product pages, navigation, cart layout, checkout messaging, subscription offers, bundles, search, filters, reviews, delivery information, product education, mobile UX, quiz flows, upsells - it goes on.

The problem is deciding what deserves attention first.

A CRO agency should be able to show how it takes research and turns it into a clear order of work. This is a huge part of the value an agency brings: knowing what do when.

At Blend, we use the PECTI prioritisation framework to score CRO recommendations before they become a roadmap.

PECTI stands for:

• Proof: what evidence supports the recommendation?

• Ease: how straightforward is it to implement?

• Cost: what resource, app cost, or development time is needed?

• Time to impact: how quickly could the change affect behaviour or revenue?

• Impact potential: what is the likely commercial value?

This stops your CRO roadmap becoming a list of opinions.

A proper prioritisation method helps you decide whether the next step should be a test, a direct implementation, further research, or no action.

What You Want to Hear

A good CRO agency should be able to explain:

• How it gathers ideas

• How it scores recommendations

• How it balances effort against potential return

• How it uses customer evidence

• How it handles internal stakeholder requests

• How the roadmap changes when new data comes in

You should be able to see the logic behind the roadmap.

If an agency can’t explain why one recommendation sits above another, you may end up paying for activity without clear commercial reasoning. Don’t be surprised if the results are underwhelming.

Answers That Should Concern You

Red flags are answers such as:

• “We usually start with the homepage.”

• “We use best practices.”

• “We will test a few things and see what happens.”

• “We’ll start wherever your team wants.”

• “We already know what works in your category.”

What you want to know is that your agency makes decisions based on a logical framework using data; all of these answers point to assumption, guesswork or a plain lack of rationale entirely.

CRO decisions need evidence, context, and clear scoring.

What Evidence Supports Your Recommendations?

Every CRO recommendation should have a reason behind it.

That reason might come from analytics, user behaviour, customer research, reviews, support tickets, user testing, or previous experiment data.

A CRO audit should identify where customers are struggling, explain why that friction may be happening, and show which fixes should be prioritised.

Blend’s CRO audit service is built around that idea: diagnosis before action.

Useful evidence can include:

• GA4 data

• Shopify Analytics

• Funnel drop-off analysis

• Heatmaps

• Session recordings

• Customer surveys

• On-site polls

• Product reviews

• Customer support queries

• Search data

• User testing

• Previous A/B test results

• Competitor and category research

Anyone can have an opinion like:

“The subscription section needs to be clearer.”

But a useful recommendation based on analysis would be:

“Mobile users are scrolling past the subscription selector and returning to it later. Reviews suggest customers value flexibility, but that benefit is not visible near the buying decision. We recommend moving subscription savings and flexibility messaging closer to the purchase area.”

Here you actually have an irrefutable reason to trust the recommendation.

It links behaviour, customer language, and commercial intent.

What You Want to Hear

Ask the agency to talk you through an example of work they’ve done with a client, and ask:

• What evidence did you review?

• What customer behaviour did you observe?

• What was the customer problem?

• What was the target metric?

• How confident were you in this recommendation?

Then look for a clear chain of reasoning in the answer.

Answers that Should Concern You

Red flags:

• “It’s a best practice.”

• “Competitors do this.”

• “The site needed to be cleaner.”

• “We have seen this work before.”

Don’t get me wrong - some of those statements are actually useful starting points, but they are definitely not enough on their own.

What Happens When a Test Fails or Comes Back Inconclusive?

Most CRO agencies are comfortable talking about winning tests. But bear in mind that most tests fail or are inconclusive - so you need to ask about how they deal with these.

This is where you learn how disciplined the agency is.

Inconclusive tests are common. Some tests don’t reach statistical significance, some produce mixed results or no change, and some reveal that the original hypothesis was wrong.

That does not automatically mean the work has failed, or that it was a bad idea.

The main thing is what the agency learns from it.

A good CRO programme documents what happened and uses the learning to improve the next decision.

Blend’s A/B testing library shows how individual tests can build a wider understanding of customer behaviour across PDPs, collection pages, cart, subscriptions, mobile UX, and revenue per visitor.

What You Want to Hear

Ask the agency:

• How do you write hypotheses?

• What is the primary metric for each test?

• What secondary metrics do you track?

• What guardrail metrics do you use?

• How long should a test run?

• What happens if a result is inconclusive?

• How are learnings documented?

• How do those learnings affect the roadmap?

A good agency should be comfortable explaining losing tests, flat tests, and uncertain tests.

If it only talks about winners, you are getting an incomplete view of the work.

Answers that Should Concern You

Be cautious if the agency says:

• “We only run tests we know will work.” (Then why test?)

• “If a test fails, we move on.” (Look for learning)

• “We guarantee a certain number of successful tests.” (You can’t)

The last answer is especially concerning, for the same reason you should be wary of overall guarantees. If an agency is under pressure to show a fixed number of winners, the reporting can become less objective.

Our own average win rate across all A/B tests April 2025 to April 2026 was 59%, which we’re proud of. But we go into depth with clients about the losing and inconclusive tests to make sure no learning gets left behind.

How Do You Decide Whether to Test, Implement, Research, or Reject an Idea?

Some CRO agencies push everything into A/B testing, which can be wildly inefficient.

A/B testing is useful when there is enough traffic (we have a minimum threshold of 50,000 sessions per month), a meaningful change, a clear hypothesis, and a decision to make.

For example, if delivery information is missing, just add it directly - treat it as a fix.

If users are dropping out at a stage where the cause is unclear, further research may be needed before any test is designed.

If an idea is based only on a competitor screenshot and has no customer evidence, it may need to be rejected.

We cover this decision-making process in more detail here: When to A/B Test vs Implement CRO Changes.

What You Want to Hear

A good CRO agency should be able to categorise recommendations into:

• Test

• Implement

• Research

• Reject or defer

And then explain the reason behind each choice, keeping your roadmap healthily prioritised.

Answers that Should Concern You

Be careful with answers such as:

• “We test everything.” (Why?)

• “We run a fixed number of tests every month.” (Some tests are much more complex and high-impact than others)

• “More tests means more progress.” (Exactly how?)

• “We don’t implement changes unless a test proves them.” (If it’s broken, fix it)

How Will You Understand Our Brand and Customer?

If the agency doesn’t understand your brand or specific sector, you’ll be getting generic CRO. Which might get you so far, but will likely plateau quickly. This is a common problem.

The agency applies familiar patterns: trust badges, sticky add-to-cart buttons, urgency messages, review blocks, cart upsells, and subscription nudges. All worthy ideas, and they could help.

But done without nuance, they might equally make the site feel less specific to your brand.

You don’t hire a CRO agency just for best practice; they should be helping you to understand why customers buy, why they hesitate, what they value, and what the brand should protect.

Ask how the agency will learn:

• Your positioning

• Your customer objections

• Your product value

• Your tone of voice

• Your category norms

• Your claims and compliance limits

• Your pricing strategy

• Your subscription model

• Your margin constraints

• Your customer support themes

• Your review language

This is especially important in eCommerce because different categories have very different buying journeys.

A customer buying coffee doesn’t make decisions in the same way as a customer buying a high-end road bicycle.

Example From Blend’s Work

In our CRO case study with snack brand Jackson's, the original brief focused on cart optimisation.

After a detailed audit, the bigger opportunity was found across the journey: product discovery, PDP structure, subscription positioning, and revenue-weighted testing.

The client came with a specific area of concern, but the audit showed a wider set of friction points.

In our CRO case study with PerTronix, the challenge was very different. Customers needed confidence that the product would fit their vehicle. In that context, CRO had to focus on clarity, compatibility, and trust.

And in our CRO work with Stone Creek Coffee, customers had to make several choices before purchase, including roast, grind, format, and subscription. We focused on reducing decision friction, particularly on mobile.

These examples show why brand and category context are important, and should shape the work your agency does.

How Will Communication and Reporting Work?

This is one of the most common frustrations with agencies of all disciplines. You need to know how often you will hear from the agency, who owns the relationship, and how decisions will be made. I cannot tell you how many times I’ve spoken to ecommerce operators with gripes along the lines of “We just wouldn’t hear from them for weeks,” “We didn’t know what they were doing,” or “Communication just wasn’t great.”

Ask specific questions:

• Who is our main point of contact?

• Who owns the CRO roadmap?

• How often will we meet?

• What is covered in those meetings?

• How are results reported?

• How are recommendations approved?

• How are blockers handled?

• Where are decisions documented?

• How quickly do you respond when there is a problem?

• What happens when priorities change?

“Monthly reporting” as a proposal line item does not cut it.

You need to know what the report includes and how it helps the business decide what to do next. A useful CRO report should explain:

• What was reviewed

• What was changed

• Why the change was made

• What happened after launch or testing

• What was learned

• What decision should follow

• How the roadmap has changed

This is especially important during ongoing CRO implementation, where research needs to become live improvements, tests, and documented learning.

Who Actually Does the Work?

CRO involves several skills, and you need to know who’s responsible for each one. Ask who handles:

• Strategy

• Data analysis

• Customer research

• UX review

• Copywriting

• Design

• Development

• A/B test setup

• QA

• Reporting

• Roadmap management

This question is important because agency delivery models vary.

Some teams provide strategy only. Some handle design and development. Some work with your internal developers. Some outsource parts of the work.

Any of those models can work, but accountability needs to be clear.

If the agency recommends a test, who designs it?

If the test is built incorrectly, who fixes it?

If tracking is wrong, who spots it?

If your internal team is implementing changes, who checks the final work?

Blend’s Shopify CRO services are designed around the connection between strategy, implementation, testing, and reporting. That connection is important because CRO recommendations only create value when they are implemented properly. A deck full of recommendations is great to have, but on its own it doesn’t achieve anything. This is exactly why we do what we call full-stack CRO.

Which Metrics Do You Optimise Beyond Conversion Rate?

Conversion rate is important, but it should not be the only number in the conversation.

A site can increase conversion rate while creating lower-quality revenue.

For example, heavy discounting may increase orders but reduce margin. Aggressive urgency may lift short-term conversion but damage trust. Pushing the wrong products may improve first purchase while hurting repeat purchase.

Ask the agency which commercial metrics it uses and why.

Useful metrics may include:

• Revenue per visitor (always our North Star)

• Average order value

• Add-to-cart rate

• Checkout completion rate

• Subscription uptake

• Subscription revenue per visitor

• Returning customer rate

• Customer lifetime value

• Refund rate

• Gross margin

• Product mix

• Repeat purchase rate

For Shopify brands, revenue per visitor is often a useful measure because it connects conversion rate and order value, though it still needs context.

A higher revenue per visitor is more valuable when it comes from better product discovery, clearer buying decisions, better subscription positioning, or healthier order quality.

At Blend, we think about growth through the Buy Trifecta:

• Buy Now: helping more of the right visitors purchase

• Buy More: increasing order value where it makes commercial sense

• Buy Again: improving subscription, retention, and repeat purchase behaviour

This helps keep CRO connected to the full customer journey.

Ask the agency how it will measure success for your business specifically.

Can You Show Case Studies that Match Our Type of Problem?

And don’t just ask whether the agency has worked in your industry. That doesn’t necessarily tell you if they’ve worked with a client that really looks like you. Instead, ask whether it has solved your type of problem.

Two brands in the same category can have very different CRO challenges.

One may have a product discovery issue. Another may have unclear subscription messaging. Another may have low mobile conversion. Another may have a technical product where customers need more confidence before buying.

Ask for examples related to your friction points.

For example:

• If customers struggle to choose the right product, ask for product discovery case studies.

• If PDP conversion is poor, ask for product page examples.

• If subscription uptake is low, ask for subscription testing examples.

• If mobile conversion is low, ask for mobile-specific work.

• If your product is technical, ask how the agency handles confidence and compatibility.

• If you want ongoing optimisation, ask for long-term examples rather than one-off test results.

Blend’s Shopify case studies and A/B testing library are useful examples of how to assess this.

When reviewing any CRO case study, look for:

• The original business problem

• The customer friction identified

• The evidence used

• The recommendation

• The result

• The learning

• The next decision

What Are You Not Promising?

This is one of the most useful questions in the sales process.

Ask the agency what it cannot promise. A credible answer may include:

• We cannot promise every test will win

• We cannot promise a fixed percentage uplift

• We cannot fix poor traffic quality through UX changes alone

• We cannot produce reliable A/B test results without enough traffic

• We cannot fully control seasonality, stock issues, pricing, or paid media

• We cannot move quickly if approvals are slow

• We cannot make a poor offer attractive through layout changes alone

That type of answer should give you more confidence - they’re being honest, and it shows the agency understands the limits of CRO.

The agency should be accountable for the quality of the work it controls:

• Research quality

• Prioritisation

• Hypothesis structure

• Test setup

• Implementation quality

• QA

• Reporting

• Learning

• Roadmap management

What Should the First 30 to 90 Days Look Like?

Ask the agency to explain what happens after you sign. The answer should be specific, and it should be easy for them to answer.

For an established Shopify brand, the first phase will often include:

• Analytics review

• Funnel analysis

• Heatmap and session recording review

• Heuristic review

• Customer research

• Review mining

• User testing where appropriate

• Competitor and category review

• Tracking checks

• Prioritised roadmap creation

• Initial implementation or testing plan

The exact process depends on whether you need a one-off CRO audit, ongoing CRO implementation, or a complete optimisation programme.

The early phase should produce a clear output.

You should know:

• What the agency will review

• What access it needs

• What will be delivered

• How recommendations will be prioritised

• When implementation begins

• What decisions your team needs to make

Be cautious if the first month is described only as “discovery” with no clear deliverable.

Discovery is valuable, but it still needs a practical outcome.

Questions to Ask Yourself Before Hiring a CRO Agency

The agency is only one part of the relationship.

Before speaking to agencies, get clear on your own situation.

Ask:

• Do we have enough traffic for reliable A/B testing?

• Is our traffic quality good enough to optimise?

• Which metric is most important right now?

• Do we need diagnosis, implementation, or both?

• Who will approve CRO decisions internally?

• Do we have development capacity?

• Are there brand rules the agency must follow?

• What has already been tested?

• What customer objections do we already know?

• Are there commercial constraints the agency needs to understand?

This helps avoid a mismatch.

For example, if your traffic volume is too low for reliable A/B testing, a test-heavy retainer may be the wrong approach. User testing, customer research, heuristic review, and direct implementation may be more useful at that stage.

If your internal team is slow to approve changes, that also needs to be accounted for in the roadmap.

CRO works best when the agency has access, context, and decision-making momentum from the client.

How to Compare CRO Agencies

When comparing CRO agencies, don’t just g on how polished the sales process feels. Look for evidence of operating discipline.

A good CRO agency should be able to show:

• A clear research method

• A prioritisation framework

• A process for deciding what to test

• A process for inconclusive tests

• A method for understanding brand and customer context

• Clear implementation ownership

• Reliable QA

• Commercial reporting

• Relevant case studies

• Honest limits on what it can promise

A flawed CRO agency will often rely on:

• Generic best practices

• Fixed uplift promises

• Vague reporting

• Too much focus on test volume

• Limited customer research

• Case studies with little detail

• Recommendations without evidence

• Poor explanation of failed or inconclusive tests

Sales bravado will seem convincing at first, but what you really want to see is whether the agency can explain its work in a way that feels specific, practical, and grounded in experience.

The Big Questions to Take into a CRO Agency Sales Call

Use this list in your next CRO agency sales call:

• How do you prioritise what to work on first?

• What evidence supports your recommendations?

• What happens when a test fails or comes back inconclusive?

• How do you decide whether to test, implement, research, or reject an idea?

• How will you understand our brand, product, and customer?

• How will communication and reporting work?

• Who will actually do the strategy, design, development, QA, and reporting?

• Which metrics do you optimise beyond conversion rate?

• Can you show examples that match our type of buying friction?

• What are you not promising?

• What should the first 30 to 90 days look like?

• How will learnings be documented and used over time?

You should leave the conversation understanding how the agency thinks, and there shouldn’t be any hint of vagueness about how they do things or what they’ll deliver.

Ready for CRO?

If your Shopify store has traffic but you are unsure what to fix first, a structured CRO audit is usually the right starting point.

If you already have insights and need a team to turn them into tests, improvements, and ongoing learning, explore Blend’s Shopify CRO implementation service.

You can also read more about why Blend works differently as a Shopify CRO agency.