Measurement Design
Part of Incrementality and media experiments
Choosing a suitable geography for a media test
Check targeting, outcome data, pre-test patterns, spillover and precision before choosing areas for a media experiment.
Choose geographic units where the planned media change can be delivered, the business outcome can be assigned consistently, and treatment and control remain meaningfully separate. A familiar city or sales territory is not automatically a sound test unit.
Check the boundaries
Start with the smallest areas the media supplier can reliably target or exclude. Then set how the outcome is assigned: by customer address, store, service location or another stable rule. Media and outcome boundaries must describe a comparison that can be interpreted.
A store catchment may suit an Australian retailer, but shoppers may cross catchments and online orders may be recorded elsewhere. Check those patterns before choosing the unit. Review campaigns outside the proposed test, and media whose coverage does not follow the selected boundaries.
Screen the eligible areas
Compare candidate areas using the same historical outcome definition and dates. Examine outcome volume and trends, not just total sales.
Note promotions, stock gaps, store openings and unusual weeks that could make a match misleading. Historical similarity helps design the comparison; it does not establish a campaign effect.
Consider likely travel and media spillover. Larger areas may reduce boundary crossing but leave fewer units for comparison. Ask whether the design can detect an effect small enough to matter to the investment decision.
| Check | Warning sign |
|---|---|
| Media control | Targeting or other campaigns undermine the assigned change |
| Outcome location | Orders cannot be assigned by a stable rule |
| Pre-test pattern | A match depends on an unusual period or diverging trends |
| Separation | Exposure often crosses the proposed boundary |
| Decision value | The detectable effect is larger than the effect worth acting on |
Key checks before selecting a geography for a media test
- Media controlTargeting or other campaigns do not undermine the assigned change
- Outcome locationOrders can be assigned by a stable rule (e.g., customer address, store location)
- Pre-test patternA match does not depend on an unusual period or diverging trends
- SeparationExposure does not frequently cross the proposed boundary
- Decision valueThe detectable effect is small enough to matter to the investment decision
Select and record the design
Consider the full eligible pool before choosing areas. Random assignment among suitable units helps avoid selecting a favourable-looking market.
With few or very different units, a matched-market approach may be more practical, but its assumptions should be stated. Where data permit, assess candidate designs on pre-test periods beyond those used to select the match.
Before launch, record area definitions, exclusions, assignment, outcome rules and the media change. If the design is too noisy or exposure cannot be separated, revise the unit or the decision question. Any later result will apply most directly to the areas, dates and change tested.
Design considerations for geo experiments in Australia
- Smallest reliable unitStart with smallest areas media suppliers can target (e.g., postcode, local government area)
- Random assignmentPreferred when feasible to avoid bias; ensures fair comparison between treatment and control
- Matched-market approachUseful with few or very different units; requires clear assumptions and pre-test validation
