Insight

Recommendation Performance Could Not Be Attributed

Stone Creek Coffee was considering whether Rebuy still justified its place in the store’s technology stack, but the available reporting could not separate results from Rebuy and Shopify’s native recommendation logic. That made the decision hard to base on commercial evidence. Product recommendations can support the choice already being made on a PDP when they are relevant; poor matches add another decision to the page. Blend therefore treated the recommendation engine itself as the variable. This followed the same evidence-led approach used across Stone Creek Coffee’s wider CRO programme and our other Rebuy-powered CRO tests: keep the customer-facing design consistent, compare each engine against meaningful buying metrics, and let observed behaviour determine which technology earns a permanent place.

Jump to results

A/B Test

Rebuy vs. Shopify PDP Recommendations

We ran an A/B test on Stone Creek Coffee’s product pages using Intelligems. The control showed ten recommendations from Shopify’s Product Recommendations API. The variant showed ten recommendations through Rebuy, using Rebuy’s own data source, while retaining the same design and position below reviews. We hypothesised that Rebuy’s recommendation logic would surface more relevant products and improve commercial performance, with conversion rate as the primary goal. Revenue per visitor, subscription performance, average order value and add-to-cart rate provided the supporting view.

The Results

The Rebuy treatment increased conversion rate by 23% and revenue per visitor by 25%. Subscription revenue per visitor rose 84%; add-to-cart rate increased 12%, average order value rose 2%, and abandoned checkout rate fell 34%. Together, the results suggest Rebuy’s recommendation logic helped more shoppers find a relevant additional product without weakening purchase intent. The engine was the meaningful variable because both experiences showed ten recommendations in the same position and design. The experiment included 3,452 visitors over four days. Intelligems reported an 89.5% probability that Rebuy beat the control on conversion and an 83% chance it was best on revenue per visitor. The test ended earlier than planned, so continued monitoring was appropriate. Stone Creek Coffee approved the Rebuy variant, Blend implemented it permanently, and the treatment went live. A useful follow-up would examine new and returning visitors separately to test whether behavioural history affects recommendation relevance.

  • 23%

    increase in conversion rate

  • 25%

    increase in revenue per visitor

  • 84%

    increase in subscription revenue per visitor
Book a CRO discovery call

See More A/B Tests