What is incrementality?

In marketing, incrementality is the marginal contribution, or the true impact, any specific marketing effort has on driving purchases, leads, or any other important customer action. If a sale is incremental, it means it wouldn't have happened if the marketing did not take place. Incrementality testing answers the question: did the marketing cause the customer to purchase or would they have purchased anyway?

Why is measuring incrementality important?

The foundation of measurement for digital marketing is attribution. This methodology is used by ad channels like Meta, Google, and TikTok for their reporting as well as by measurement tools like Google Analytics. Attribution observes marketing touchpoints, clicks and views when possible, at an individual user level, then assigns credit for each purchase to the marketing touchpoints.

Attribution has three major limitations:

1. Collecting every touchpoint at a user level across all channels is challenging. This is for a variety of reasons and continuously becoming more limited.

2. Attribution doesn't measure causation, just correlation.

3. Ad channels' self attribution only considers marketing touchpoints that happened within their own platform, lacking the full picture of your marketing channels.

Incrementality tests complement attribution as they measure what attribution can't, causation. Incrementality tests are a way to audit your attribution and validate your measurement.

How do you measure incrementality?

Incrementality can be measured by running a controlled trial, similar to any scientific experiment run across any industry. For incrementality tests, the experimental group (the test), gets shown ads over a period of time. While the control group doesn't get shown ads for that same period. Both groups' purchases, revenue, or other key metrics for the time period are then compared. The difference between the two groups is the incremental impact of the ad channel or tactic that is being measured.

How do you divide control and exposed groups for incrementality tests?

There are two ways to divide audiences:

1. Randomly assigned audiences - This is the standard in scientific controlled trials, but is not really practical for advertising. The ability to split the testing population at a user level into two randomly assigned audiences is not available for most channels and is often impossible to truly execute as complete user level identity resolution is needed to accurately measure incrementality using this method.

2. By geography (Geo-based Match Market Incrementality Test) - For geo-based match market incrementality tests, also known as geo-lift testing, individual geos (defined at DMA, ZIP code, or other level) are divided into two comparable groups based on the historic performance of each geo. This approach is the best to use as it's both statistically rigorous and feasible to execute.

How to setup an incrementality test?

Step 1: Pick what channel or tactic you want to test

This can be at the channel level, like Meta ads, or a subset level like Performance Max. The feasibility of what level of granularity can be tested is based on available sample size. So testing at a very granular level, such as ad, is not practical nor likely possible.

Step 2: Define control and test geo pairs

This is done by analyzing past purchasing behavior of individual geos and creating two groups of geos with similar historical performance. This is a complex analysis and a data science background is needed to perform it.

Step 3: Define additional test details

This includes the length of the test. Again, a data scientist will determine this based on multiple factors.

Step 4: Build the test in the ad channel

Build the test in the ad channel according to the test details determined in the steps above. The test will run for a predetermined time period.

To properly run ongoing incrementality tests, you need a robust team of data scientists and data engineers working together with your marketing team. While you can do this in-house, it's rarely cheaper or faster to do so. Solutions like WorkMagic automate the whole process, removing the need for internal data science and data engineering resources.

Because incrementality tests are a snapshot in time, our learning shows that you should retest each channel every 3 months to have the most accurate measurement.

How do you calculate incremental lift?

After your test concludes, you calculate the difference between the exposed and control group for the metrics you care about. For incremental lift the calculation is:

((Test Purchases - Control Purchases) / Control Purchases) x 100 = Incremental Lift*

For example, if there were 140 purchases in the test geos and 100 purchases in the control geos, the calculation would be:

((140 - 100) / 100) x 100 = 40%

So the test geos had a 40% incremental lift over the control geos.

*This is a simplification of the equation. WorkMagic uses an advanced statistical methodology for its actual calculation.

How do you calculate incrementality?

The incremental impact of the media, or its incrementality, is calculated similar to incremental lift, but test purchases are used for the denominator:

((Test Purchases - Control Purchases) / Test Purchases) x 100 = Incrementality*

Using the same example with 140 purchases in the exposed geos and 100 purchases in the control geos, the calculation would be:

((140 - 100) / 140) x 100 = 29%

*This is a simplification of the equation. WorkMagic uses an advanced statistical methodology for its actual calculation.

How do you calculate incremental return on ad spend (ROAS)?

The calculation for incremental ROAS is:

(Incremental Revenue / Ad Spend)

For example, if you spent $100,000 on ads for the test geos and had $150,000 in incremental revenue, the calculation would be:

($150,000/$100,000) = 1.5x incremental ROAS

How do you optimize based on incrementality test results?

Next up, is optimizing based on the findings. Remember, incrementality tests are run one channel or tactic at a time at an aggregate level, so knowing how to apply the learnings to optimize your media is not as straightforward compared to using multi-touch attribution. To solve this at WorkMagic, we take the results of your incrementality tests and combine it with our data driven attribution model, giving each brand a custom incrementality adjusted attribution model. This model gets applied to all your dashboards across all levels of granularity, from business level to ad level, making optimizing for incrementality simple.

This methodology allows marketers to optimize for incrementality in real-time at every level of granularity, just like they would with a regular multi-touch attribution model. WorkMagic makes running incrementality tests simple for marketers, so they don't need to rely on data science resources, reducing the cost and speeding up the testing process.

What channels should you test for incrementality?

It's important to measure incrementality for channels and tactics that you can feasibly test. A channels feasibility will be based on:

1. Having enough volume to test.

2. Being able to control targeting at a geo level.

A brand should start by testing their largest channels, especially ones they believe that their attribution model doesn't properly credit. For most advertisers, testing starts with Google and Meta ads, or other major ad channels such as TikTok or CTV ads.

What channels are difficult to measure incrementality?

Channels with insufficient volume levels and/or unable to control targeting at a geo level are not a fit for incrementality testing. Podcasts and affiliate channels are examples of channels that are difficult to control targeting at a geo level. For these channels, incrementality can be measured using less accurate methods such as on/off tests or modeling.

What are common concerns with incrementality testing?

A common concern is lost revenue from the holdout. The holdout group is usually 10-30% of the total audience of the channel being tested. At a whole business level, this normally only equals about 1-3% of overall revenue, with the decrease in profit being even less as testing reduces your ad spend cost. While there may be lost revenue in the short term from decreasing your marketing spend, the long-term impact of measuring and optimizing to incrementality will greatly outweigh it.

What are the limitations of incrementality testing?

• Incrementality tests are only feasible if your business has a large enough sample size to be able to create matched geo pairs. A rule of thumb is that a business needs 3,000 or more orders a month to be able to run geo incrementality tests. The exact number of orders needed is determined by an analysis of a company's purchase data.

• Incrementality tests are a snapshot in time. This is why incrementality testing shouldn't be a one-time thing, but an ongoing process to better understand your media's true impact.

• Incrementality tests are on an aggregate level and don't provide measurement at the most granular levels.

  
  

Conclusion

Measuring for incrementality has never been more important, or easier to do. Brands that optimize for incrementality measure for causation, restore the signal strength of their data, and overcome the natural limitations of standard multi-touch attribution. For brands with sufficient volume, incrementality testing should be a foundational component to their measurement solution.

If you want to get started with testing and optimizing for incrementality, schedule time with a WorkMagic Measurement Expert to learn more about how your business can leverage incrementality testing to make better decisions.