Incrementality and experiments: holdouts, geo tests and lift, designed and read honestly

Incrementality is what an activity added: the difference between what happened and what would have happened without it. See incrementality in the glossary. Only an experiment, with a control group chosen at random, shows the second half of that sentence. The foundations piece measuring effectiveness lists the designs; this piece is about running them well.

• The decision it informs whether a channel, a campaign, a spend level or a creative choice actually causes the outcome, before money moves on the strength of it.
• The designs user-level holdouts, geo experiments that randomise whole regions¹, and platform lift studies that use randomised control groups^6, 7.
• The hard part advertising effects are small relative to the noise in sales, so many tests cannot detect them; Lewis and Rao found a median confidence interval on return on investment more than 100 percentage points wide across 25 large experiments⁴.
• Why it matters observational methods often fail to reproduce experimental results, even with rich data⁸.
• Its place experiments establish cause and calibrate the mix model and attribution⁹.

Every plan rests on a few large assumptions: that paid social adds new buyers, that television still drives search, that retargeting is worth its cost, that doubling spend in a region will lift sales. An experiment turns one such assumption into evidence. The IPA's Making Effectiveness Work places experiments second in its MESI loop: the model points to where the plan could change, the experiment tests that change, and the results feed simulation, implementation and the recalibration of attribution and the model⁹.

The discipline is to test what would change a decision. A test whose result nobody would act on is a cost, not a learning. Write the decision first ("if the lift is below X, we cut the channel by a third"), then design the test that can tell you.

Holdouts. A random share of the audience is kept out of a campaign, a channel or a message stream, and outcomes are compared. They are natural for channels you deliver yourself, such as email and CRM programmes, where you decide who receives what. A standing holdout, a small share of customers who never receive a given programme, lets a CRM team measure that programme's contribution continuously rather than once.

Geo experiments. Non-overlapping regions are randomly assigned to test or control, and each region experiences its condition through geo-targeted advertising. Vaver and Koehler at Google described the approach in 2011 as conceptually simple, with a systematic design process and results that are easy to interpret¹. A follow-up paper rotated test and control across several periods to produce a series of measurements rather than a single read². Where there are few regions to work with, as in smaller countries or matched-market tests, Wang and Vaver's time-based regression predicts the counterfactual sales series for the test regions from the control regions and estimates the cumulative effect at the end of the test³. Geo tests measure real sales and need no user tracking, which makes them robust to the consent and cookie changes described in attribution and its limits.

Platform lift studies. Ad platforms run conversion lift and brand lift studies with randomised control groups. A key method behind them is the ghost ad: in the control group, the system records when it would have shown the ad, so exposed people can be compared with their true counterparts. Johnson, Lewis and Nubbemeyer report that this measures lift about as precisely as older designs (public service announcement controls and intent-to-treat tests) while costing at least an order of magnitude less⁶; the American Marketing Association noted when honouring the paper that several advertising platforms offer ghost ad methods⁷. The limit is that each study sees one platform, run by the platform.

The most common failure is a test too small to see anything. Lewis and Rao showed why: individual-level sales are very volatile relative to the cost of advertising per person, so informative experiments can need more than 10 million person-weeks, which may put them out of reach for many firms⁴. Their broader survey of digital measurement with David Reiley makes the same case for careful design⁵. In practice:

• Measure closer to the ad when sales are too noisy, but say so: a lift in site visits is not a lift in profit.
• Prefer regions when individuals are untrackable and the product sells offline.
• Run long enough to cover the purchase cycle and any carryover, and to allow for a week or two of ramp-up.
• Agree the threshold in advance A result is "positive" only against the threshold set before launch.

Experiments are unbiased, but not precise by default. Lewis and Rao conclude that randomised trials still add unbiased information even when intervals are wide⁴; the honest report says what the test ruled out and what it did not. A test that cannot distinguish "no effect" from "a large effect" has answered nothing about the budget, and the right response is a bigger or longer test, not a confident headline.

Read the result for what it is, too. A test measures one activity, in one period, in the audiences or regions tested, at the spend level tested. It does not say what happens at double the spend, in another season or with another creative, and it usually misses effects that arrive after it ends, which is why brand effects rarely show in a short conversion test. Treat each result as one point on the response curve the mix model is trying to draw, not as the curve itself, and repeat the important tests: Google's design that rotates test and control over several periods was built for that kind of ongoing measurement².

An experiment answers one question at one time. Its greatest value comes when it calibrates the other methods. Robyn's description includes optional calibration with ground-truth experiment results¹⁰; Bayesian models such as Google's start from priors, which carry most weight precisely when data are thin, so a well-designed experiment is the best prior a model can get¹¹. Gordon and colleagues' comparison of experiments with observational methods on the same campaigns is the reason to prefer that ground truth⁸. Attribution rules should change too: if a holdout shows that retargeting adds little, its credit in the daily report should be discounted accordingly.

Across this track, the triangulation runs: the mix model proposes, the experiment tests, attribution is corrected, and the brand tracker shows whether the long-term picture moves with it.

Illustrative, with no real data. A subscription service spends heavily on paid social and believes it brings new customers. The mix model cannot separate it from seasonal demand, because spend has always risen in the same months. The team writes the decision: if the incremental cost per new subscriber is above its target, paid social is cut back to a maintenance level. It splits the country's regions into matched pairs, randomly assigns one region of each pair to pause paid social for eight weeks, and fixes the analysis in advance. The result shows a real but smaller lift than the platform's own report, with a range that still sits above the cut-off. The team keeps paid social at a lower level, feeds the result into the model as a prior and schedules the next test for a creative change.

1. Testing what nobody will act on Start from the decision.
2. Underpowered tests Too few people, regions or weeks to see a realistic effect⁴.
3. Peeking and stopping early Decide the duration in advance.
4. Reading a platform lift as cross-channel truth It measures one platform, run by that platform.
5. Changing other things mid-test A promotion in the test region ruins the comparison.
6. Filing the result A test that does not update the model and the attribution rules is wasted.

On Monday:

1. List the three biggest assumptions in this year's plan and the decision each one drives.
2. For the top one, pick the design with the ladder above and write the decision rule.
3. Size the test with your analyst or agency: the smallest effect worth knowing, and the sample and weeks it needs.
4. Write the analysis plan before launch and share it with finance.
5. Keep test and control clean; log anything that happens in either.
6. Report the range and what it rules out; then update the mix model and the attribution rules.
7. Keep a learning agenda, a running list of questions and results, alongside the measurement framework, and review it with the CMO each quarter.

• Vaver and Koehler on geo experiments^1, 2; Wang and Vaver on time-based regression³.
• Lewis and Rao on the economics of measuring advertising⁴; Lewis, Rao and Reiley on the digital frontier⁵.
• Johnson, Lewis and Nubbemeyer on ghost ads^6, 7; Gordon and colleagues on experiments and observational methods⁸.
• The IPA's Making Effectiveness Work⁹; Robyn's calibration¹⁰ and Google on Bayesian priors¹¹.
• On IMC Report: measuring effectiveness, the measurement objective, and the glossary entries for A/B test and brand lift.