Launch-to-Measure Testing: Measuring incremental effects with smaller test budgets
by
Daniel Oh
How "pause-to-measure" and "launch-to-measure" tests work
Most geolift tests, including the ones at WorkMagic, work off a methodology we call "pause-to-measure". The way it works, in an extremely basic way that annoys our Head of Data Science, is this:
You set aside two statistically similar groups of geos - a larger exposed group and a smaller holdout group.
You pause ads to the holdout group and continue showing ads to the exposed group.
Measure the difference in behavior between the holdout and exposed groups.
Recently, we released another methodology we call launch-to-measure, which flips the ratio between exposed and holdout groups. The way it works (again, by stripping out a lot of the particulars) is this:
You set aside two statistically similar group of geos - a larger holdout group and a smaller exposed group
You show ads to the exposed group and leave the holdout group alone
Measure the difference in behavior between the exposed and holdout groups.
It's important to understand that both tests are fundamentally the same: they isolate similar geo pairs from the larger national audience, then chooses one group to run business-as-usual while the other group gets a different treatment. They'll return results with the same level of confidence and statistical significance. The main difference is how each test type sets up ad exposure:
Pause-to-measure tests the impact of pausing ad operations on a small holdout sample.
Launch-to-measure tests the impact of introducing ads to a small exposed sample.
Pause-To-Measure (PTM) | Launch-To-Measure (LTM) | |
|---|---|---|
Cell 1 - Exposed to Ads | Statistically similar geos that make up 5-10% of sales | Statistically similar geos that make up 5-10% of sales |
Cell 2 - Held Out From Ads | Statistically similar geos that make up 5-10% of sales | Statistically similar geos that make up 5-10% of sales |
Rest of the Country (Business as Usual) | The remaining 80-90% of your sales that are shown ads as usual | The remaining 80-90% of your sales that aren't shown new ads as usual |
Spend Required | Higher - most of the audience is exposed | Lower - only a small sample is exposed |
Which methodology should I use, and when?
There are two main things to consider: how will this test affect daily operations and how much budget do I have to spare?
A good lift test minimizes disruption. In order to determine which test methodology to use, you have to ask yourself why you're testing in the first place.
If the channel, tactic, or campaign you're testing doesn't have a significant budget assigned to it, launch-to-measure tests work best because you only need a fraction of your existing budget to find meaningful lift. That way, your established channels can keep running as ordered.
If the channel, tactic, or campaign you're testing does have a significant budget assigned to it, then you'd want to keep most of it running normally. This is where a pause-to-measure test fits, because you only need to set aside a fraction of your audience as a holdout, while the rest of your audience gets their normally scheduled ad exposure.
Testing your testing limits
The beauty of launch-to-measure tests isn't only because they're cost-effective, they remove barriers to lift testing in general. Having multiple options on hand gives marketers more possibilities to run tests, even during high-leverage moments where budgets, seasonal demand, or expectations fluctuate.
So leverage our launch-to-measure tests to fill up your experimentation calendar. The results you get from either a pause-to-measure or launch-to-measure will continually give you the causal direction you need to continue optimizing towards your key business goals.
Lift testing at WorkMagic
At WorkMagic, you can set up either type of test without exhausting setup times and implementation cycles. If you're new to the platform, we'll walk you through your initial tests, making sure we consider every factor that could affect the strength of the results. Afterwards, our self-serve platform makes it easy to design and launch lift tests on a regular cadence.
Of course, lift tests are great at capturing snapshots of your marketing efforts and validating your hypotheses. But as time passes, the results become less actionable. Markets, buying behavior, and trends can all shift in the space of a month.
What we do at WorkMagic is different, we use the results of your lift tests to calibrate brand-specific MTA models and MMM projections that inform your short-term and long-term decision making. There's nothing more accurate and grounded in truth than the impact measured in a well-designed lift test. So why wouldn't you want to use those results to inform your other marketing measurement models?
Want to learn more? Reach out and schedule a demo with us!