The Retargeting Fallacy: A Case Study in Incrementality vs. Attribution
A luxury footwear retailer ($1,200 AOV) showed me their retargeting dashboard. ROAS was 8x. They wanted to scale it.
I told them their retargeting was probably losing money. They didn't believe me. So we ran the test.
The problem with retargeting ROAS
Standard attribution (especially last-click) incentivizes targeting high-intent users. Someone visits your site, browses for 20 minutes, adds to cart, leaves. You retarget them. They come back and buy. The ad gets credit.
But would they have bought anyway?
For a $1,200 shoe with a 2-4 week consideration cycle, the answer is almost certainly yes. The customer was already convinced. The retargeting ad didn't change the outcome. It just inserted itself into the path and took credit.
This creates a feedback loop: the more you spend on retargeting, the better your attributed ROAS looks, because you're targeting people who were already going to convert. The platform dashboard shows a winning campaign. The actual P&L tells a different story.
High ROAS on retargeting is often a signal of selection bias, not efficiency. You're measuring touch ("did the ad exist before the sale?") when you should be measuring lift ("would the sale have happened without the ad?").
How we set up the holdout test
To isolate actual causal lift, we ran a randomized controlled trial using Meta's audience segmentation.
Step 1: Define the holdout. We split the retargeting audience into two groups using Meta's Audience Manager. Treatment group (90%) saw retargeting ads as normal. Control group (10%) was suppressed from all retargeting.
The 90/10 split wasn't statistically ideal. 50/50 would have been better. But the client wouldn't risk turning off retargeting for half their warm audience. The 90/10 was the political compromise.
Step 2: Set the test window. We ran for 14 days. Their historical data showed 80% of abandoned-cart-to-purchase conversions happened within this window. Longer would have been better for confidence, but 14 days captured the majority of the decision cycle.
Step 3: Measure conversion rate, not revenue. This is where most holdout tests go wrong. Don't compare total revenue between groups (the 90% group will always have more because it's bigger). Compare conversion rate: what percentage of each group purchased?
Step 4: Calculate incremental lift. The difference in conversion rates between the two groups, multiplied by the total audience size, gives you the number of conversions the ads actually caused.
The results
After two weeks:
- Group A (saw retargeting ads): 5.2% conversion rate
- Group B (no retargeting ads): 5.1% conversion rate
- Incremental lift: 0.1 percentage points
The retargeting campaign was generating almost zero incremental conversions. The people who saw ads converted at essentially the same rate as people who didn't.
The real economics
The platform showed 8x ROAS based on attributed sales. The incremental math:
- Ad spend: $10,000 over 14 days
- Incremental revenue: ~$800 (from the 0.1% lift on the audience)
- Incremental ROAS: 0.08x
- Net result: -$9,200
The "profitable" retargeting campaign was a $9,200 loss in two weeks. Extrapolating over the 7 years they'd been running this strategy: roughly $280,000 in budget spent acquiring customers who were already acquired.
I've seen this pattern across luxury retail, B2B SaaS with long sales cycles, and any business with high brand awareness in their primary market. The higher the organic conversion rate, the more retargeting masks itself as performance.
What we did with the budget
We shifted 70% of the retargeting budget to prospecting. New audiences, new geos, people who hadn't visited the site yet.
Attributed ROAS dropped from 8x to 2.4x. The CFO panicked for a week. Then Q1 revenue came in 18% above the same period the prior year. The lower-ROAS campaigns were finding net-new customers instead of re-showing ads to people who already had items in their cart.
How to run this yourself
You don't need a data science team. The test takes 30 minutes to set up in Meta:
- Go to Audiences in Meta Business Manager
- Create a custom audience from your retargeting pool
- Use the randomization feature (or split by last digit of user ID) to create a 90/10 holdout
- Exclude the 10% control group from all retargeting ad sets
- Run for 14-21 days
- Compare conversion rates, not total revenue
If the conversion rate difference is less than 1 percentage point, your retargeting is mostly claiming credit for organic conversions. The budget is better spent elsewhere.
The question isn't whether the ad touched the customer before the sale. It's whether the sale would have happened without the ad. If you can't answer that, you're paying for your own organic success.