Measurement Design
Part of Incrementality and media experiments
Comparing a holdout with ordinary historical reporting
See what an ordinary historical media report can show, what a holdout adds and when neither supports a confident causal claim.
Use a holdout when the decision needs evidence about what a media change caused. Use ordinary historical reporting to describe what happened and to help design the test. A before-and-after increase can also reflect changes in demand, prices, stock or other marketing.
What each comparison observes
A historical report compares the campaign period with an earlier period. It can show outcomes and spend against past values, but an ordinary before-and-after chart does not observe what would have happened during the campaign period without the tested change. Choosing the same period last year does not remove every difference between the two periods.
A holdout assigns a control group to receive a different media condition while the treatment group receives the planned change. Both are observed during the test period. When assignment, delivery and outcome measurement hold, that concurrent comparison supports a more credible estimate of the tested change. A holdout can still be inconclusive if groups are poorly comparable, exposure crosses between them or the result is imprecise.
| Question | Ordinary historical report | Holdout experiment |
|---|---|---|
| What happened? | Describes observed outcomes over time | Describes outcomes in assigned groups |
| What is the comparison? | An earlier period | A concurrent control under the design |
| Does it isolate the tested change? | Not on its own | More credibly if the design holds |
| Main practical trade-off | Competing explanations remain | Setup, measurement and potentially withheld exposure |
Put historical data to work
Inspect prior outcomes for volatility, gaps and unusual promotions. In a geographic test, examine candidate areas across pre-test periods rather than matching them on one convenient total. Agree on the business outcome and response window before assignment.
Suppose an Australian service business receives more qualified enquiries after introducing video ads. Accurate records establish the increase, but they do not show how many enquiries would have arrived anyway. A proposed holdout would need eligible groups, a stable enquiry definition and a check that the intended video delivery differed between them.
Key Considerations for Holdout Testing in Australian Markets
- Pre-test data inspection
- Check for volatility, gaps, or unusual promotions (e.g., end-of-financial-year campaigns)
- Geographic test eligibility
- Evaluate candidate areas across multiple pre-test periods, not just one total
- Outcome stability
- Ensure enquiry definitions remain consistent (critical for ABN-registered service businesses)
- Media delivery clarity
- Confirm video ads differ meaningfully between groups (e.g., placement, targeting)
Decide whether a holdout is feasible
A holdout may be impractical if exposure cannot be varied cleanly, the outcome is too sparse for a useful estimate or the business cannot accept withholding the activity. When controlled testing is infeasible, report the historical trend and its competing explanations without labelling it causal lift.
More elaborate historical modelling is a separate method with its own assumptions; it is not created by adding a baseline line to a chart.
After a holdout, historical data can still provide context for an unusual week. Keep the experimental estimate tied to the population, dates and media change actually tested.
