Measuring Virtual Try-On Incrementality
Measure try-on by orders per eligible shopper in a randomized holdout. The conversion rate among shoppers who chose to try on answers a different question now.
Measure virtual try-on by holding back a random slice of eligible product-page shoppers and comparing their orders with the shoppers who could see the tool. The result you want is incremental orders per eligible shopper. The conversion rate among people who chose to try something on answers a different question.
Exposed receipts on one side of the tape, holdout receipts on the other. The tape is the whole method. Editorial image in Classic Antla disposable-camera style.
Teams already have a number for try-on. It is usually the purchase rate among people who finished a preview. That number describes a group that raised its hand. Incrementality asks what changed for shoppers who were allowed to see the tool, including the ones who never tapped it. Shopify’s A/B testing guide is the general version of that split: one change, a control, a comparison you can defend. The rest of this page is the fashion-specific version.
The write-up of conversion engine metrics separates the 3.8% try-on-user rate from lift and from ROI. Shopify virtual try-on conversion walks the on-site path. This article is the holdout. If you only need the reason the try-on-user rate misleads, read the selection-bias note beside it.
Define eligible population
Eligible means a shopper who could have used the preview if the test assigned them to see it. Everyone else is outside the denominator, on both sides, or the holdout will look artificially good.
Include a shopper when all of these are true:
- They opened a product page where the preview is offered: supported category, in-stock variant, theme block able to render.
- They are a person you can assign, usually a first-party cookie or a logged-in customer id. A session that flips assignment mid-browse ruins the split.
- They are not a bot, an internal IP, or a preview link your team uses for QA.
Exclude, from both arms:
- Pages where the block cannot load, so “exposed” would be a broken button.
- Products you have turned the tool off for.
- Orders that happen with no product-page view in the window, such as a reorder from the account page.
Write the rule down before the test starts. Changing “eligible” after you see the chart is how a flat test becomes a win.
Decide the unit. I assign the shopper, not the session, and I keep that assignment for the whole test. The same person should not be exposed on Monday and held out on Thursday. If your stack can only randomize sessions, say so in the readout and expect the effect to shrink, because people cross arms.
Also decide the window. I use seven days from the first eligible product view: any order, and a second column for an order that contains the variant they viewed. Those are different decisions. A try-on of a coat that is followed by socks is not the same claim as a coat order.
Shopify’s conversion-rate guide makes the denominator the whole argument. Device, source, and what you count as an order all move the rate. An incrementality readout inherits that. Publish the definition in the same slide as the result.
Randomize exposure
Exposure means the shopper can see and use the preview. Holdout means the product page looks as it did without the tool: no button, no empty slot, no “coming soon.” A holdout that shows a disabled button measures frustration, not the absence of a preview.
Assign randomly at the moment of eligibility, with a fixed share. Half and half is the fastest way to learn. A 10% holdout is kinder to revenue when you already believe the tool works and you want a standing measurement. Either way, the share is set in advance.
Sticky assignment beats a coin flip on every page view. Store the arm on the shopper. If they arrive from email tomorrow, they stay in the same arm.
A clean setup on Shopify usually means the theme app block renders only for the exposed arm, while the holdout theme path is the ordinary gallery. Do the QA on a phone: exposed shoppers see the block, holdout shoppers do not, and a refresh does not swap them. Check a custom theme separately from a stock one. A block that fails open and shows everyone is no longer a test.
Do not randomize by product if the question is the shopper-level effect. Product splits confound the tool with the SKU. You can still read results by category after a shopper-level split.
Antla reports what happened among people who used a preview. That report is the activity log. The holdout is yours to build, because a vendor log cannot see the shoppers who were never offered the button.
Track order conversion
The primary metric is orders per eligible shopper in the window, exposed minus holdout. Revenue per eligible shopper is the pair to publish beside it. Contribution margin per eligible shopper is what finance should remember, after discounts and a return allowance you apply the same way to both arms.
Build the table like this. The figures are a teaching example so the arithmetic is visible. They are not a benchmark and they are not an Antla result.
| Exposed | Holdout | |
|---|---|---|
| Eligible shoppers | 18,000 | 18,000 |
| Orders in 7 days | 342 | 306 |
| Orders per 100 eligible shoppers | 1.90 | 1.70 |
| Revenue per eligible shopper | $1.71 | $1.53 |
Incremental orders are 342 minus 306, which is 36, on 18,000 exposed shoppers. The rate difference is 0.20 orders per 100 eligible shoppers. Quote both. A relative lift of about 12% and an absolute gap of 0.2 points are the same example, and readers mix them up when only one is on the slide.
Secondary columns, same arms, same window:
- Orders that contain the viewed variant.
- Revenue and contribution per eligible shopper.
- Return rate on those orders, once the return window has closed. Early readouts will undercount returns.
- Preview starts per exposed shopper, so you know the tool was actually available. A near-zero start rate means the block was buried. It does not mean the holdout “won.”
Leave try-on-user conversion off the primary row. Across more than 500,000 completed try-ons, 3.8% of try-on users purchased. The population in that rate is people who finished a preview. A holdout of eligible shoppers is not beside it, so the rate cannot name an incremental effect. Antla merchant reporting, cohort dates not published.
The 35% figure is a different statement again: try-on users converted 35% higher, on average, than shoppers who did not complete a try-on. The comparison group is non-users, and they were not randomly kept from the tool. Antla merchant reporting, cohort dates not published. Keep that association in an appendix if someone asks what the activity log shows. The decision number remains exposed versus holdout.
Shopify’s behavior reports will not compute this split for you. They will help you check that sessions and checkouts did not break during the test. Reconcile the holdout’s order count with Shopify orders for the same days before you present.
Report uncertainty and guardrails
A point estimate without a range is a mood. For the rate difference, report a confidence interval or a posterior interval your analyst will sign, and say which one. If the interval on orders per eligible shopper includes zero, the honest summary is that the test did not show a reliable order gain. You can still look at guardrails. You should not book the upside in a forecast.
Pre-register three things in the test note: the primary metric, the window, and the minimum length (I would not stop a fashion test before two full weeks, because weekends buy differently). Peeking every morning and stopping when the line looks friendly inflates the result. The same Shopify testing guide is worth reading on that discipline before the first Monday standup.
Guardrails are metrics that can veto a win:
| Guardrail | Stop or pause if |
|---|---|
| Preview failure rate | Generations error often enough that support hears it daily |
| ”Looks wrong” tickets per 100 starts | The rate jumps against the prior month on the same category |
| Checkout completion among shoppers who reached checkout | Exposed falls while holdout holds |
| Return rate, after the window | Exposed returns rise by enough to eat the margin |
| Page speed on the template | The block blows the mobile budget you already set |
The conversion engine definition is the wider system this test sits inside. Incrementality for the preview is one loop, the visit. Recovered revenue from later messages is a second test with its own holdout. Adding those orders into this readout double-counts people who would have come back anyway.
Publish a short memo: definition of eligible, assignment share, primary difference, interval, guardrails, and the decision (keep, iterate on placement, or turn off). Link the raw counts. A 12% relative story with the teaching numbers above is 36 orders. Say 36 orders.
Run one holdout on the hero template
Turn the block on for a random half of eligible shoppers on the template that already has honest product photos. Leave the other half on the ordinary gallery. Read orders per eligible shopper at fourteen days, with the interval, and ignore the try-on-user rate until the memo is written.
If the interval includes zero, the next move is placement or the offer, not a louder claim. The activity log can tell you who tried the coat. The tape on the floor tells you whether the offer changed the orders.
About the author: Aaron founded Antla. He wants the holdout box in the readout even when the exposed box is more fun to talk about.