In short
- The cleanest way to measure whether a channel actually caused conversions, rather than just took credit for them
- A control group is held out from seeing the ads; the exposed group sees them as normal
- Incremental lift = conversion rate of the exposed group minus the control group
- Meta and Google offer native conversion-lift studies; geo holdouts work for channels without them
What a holdout test is
A holdout test answers the one question attribution cannot: did this marketing actually cause the conversion, or would it have happened anyway? It works by randomly splitting your audience into two groups. The control (holdout) group is prevented from seeing a given ad or campaign; the exposed group sees it as normal. Because the two groups are statistically identical apart from exposure, any difference in their conversion rates is caused by the ads. That difference is the incremental lift.
It is the experimental backbone of incrementality measurement, and the reason a channel with a high attributed ROAS can still turn out to drive very little net new revenue.
Where holdout tests are run
- Native platform lift studies. Meta Conversion Lift and Google’s equivalent hold out a control group inside the platform and report the measured lift. Easiest to run, but you are trusting the platform to grade its own homework.
- Geo holdouts. Turn a channel off in a representative set of regions, leave the rest running, and compare. Slower and more disruptive, but platform-independent and good for channels with no native lift tooling.
- Matched-market tests. Pair similar markets, treat one and hold out the other. Statistically cleaner than a naive geo split, at the cost of careful market matching.
How to use the results
A holdout test produces an incremental ROAS (revenue lift divided by test spend). Compare it to the ROAS your attribution model reported for the same channel. When the attributed number is far higher than the incremental one, the channel is over-credited, and you can down-weight it in your model. Re-test on a cadence, because lift shifts as audiences saturate and creative fatigues.
FAQ about Holdout Tests
What is a holdout test?
An experiment that withholds a campaign from a random control group and compares their conversions with an exposed group. The difference is the conversions the campaign actually caused.
How is a holdout test different from attribution?
Attribution assigns credit to touchpoints a converter happened to see. A holdout test measures causation directly by comparing against people who did not see the ads, so it catches conversions that would have happened anyway.
How big does the holdout group need to be?
Large enough to detect the expected lift with statistical significance. Small lifts on low conversion volume need big samples; a 3% lift on 50,000 conversions is real, a 10% lift on 200 is noise.