The Hidden Cost of DIY Incrementality Testing

James Gerber

by

James Gerber, SVP of Strategy

Last updated:

Last updated:

An AI assistant like Claude can write a working geo lift script in an afternoon. Pair it with an open-source package and an export of your sales data, and running your own incrementality test looks like a weekend project.

The code is the easy part. A trustworthy test depends on a design that holds up before you spend, a clear view of whether the test can detect anything at all, a readout that survives scrutiny, and plumbing that keeps the whole thing clean. Get any of those wrong and you lose a month of spend plus a holdout, and you usually find out only at readout.

Here is where DIY tests tend to break, and what it takes to avoid it.

1. The design has to be validated before any money is spent

Most open-source geo packages lean mainly on synthetic control. It is a useful method, but it tends to overfit: the control group can match the test group’s history closely and still predict its future poorly.

A stronger approach treats market selection as an optimization problem. A constrained optimizer searches for the split whose control group best predicts the test group’s pre-period sales, while staying within limits on coverage and geographic spread. Then the chosen design has to pass a battery of stability checks, including a placebo test, before it launches.

This matters because a badly matched test fails silently. Spend goes out, the holdout gives up revenue, and the problem only shows up when the result comes back noisy or implausible.

2. You should know upfront whether the test can work

Every geo test has a minimum detectable lift: the smallest effect it can reliably tell apart from noise. That threshold implies a budget floor. Spend below it and the test was always going to come back inconclusive, no matter how well the channel performs.

Calculating both before launch turns a gamble into a decision. If the floor is out of reach, you can change the test length, the markets or the scope, or choose not to run it, instead of paying for a month of data that can’t answer the question.

Illustrative chart: minimum detectable lift falls as test budget grows; below the budget floor the test can't detect the expected lift.

3. The readout shouldn’t rest on one model

A lift estimate is only as good as the counterfactual behind it: the sales the test markets would have had without the change. A DIY setup typically produces that counterfactual from a single model and reports its answer.

A more defensible readout runs sensitivity checks on the counterfactual, so you can see whether the result holds when the modeling assumptions shift. If it does, you can act on it with confidence. If it doesn’t, you know the number is fragile before you move budget on it.

4. The result should keep working after the test ends

This is the biggest gap. An open-source tool like GeoLift gives you one number, and that number usually lands in a deck. A few weeks later, budgets are set by platform-reported ROAS again and the test is a memory.

In WorkMagic, the lift result calibrates daily attribution through iDDA (incrementality-adjusted attribution), so the credit each channel gets every day reflects what the test proved. It also feeds planning through MMM and the Media Budget Optimizer. One test keeps improving decisions for months, instead of informing a single meeting.

DIY: a lift result becomes one slide, then budgets go back to platform ROAS. WorkMagic: the lift result feeds daily attribution (iDDA), MMM and the Media Budget Optimizer.

5. The operational load is the part nobody budgets for

Running tests in-house means owning everything around the model:

  • Geo-level data pipelines. Clean, daily sales by market across DTC, Amazon, TikTok Shop and retail, kept current for every test.

  • Spillover handling. Shoppers and delivery zones cross market lines, and that leakage has to be accounted for.

  • Concurrent tests. Two tests that share markets contaminate each other, so someone has to keep them apart.

  • Mismatched platform targeting. Not every ad platform targets the same geographies. One may support DMAs while another only supports states, so the design has to balance a state-level setup on one platform with a DMA-level setup on another.

  • Data science time on every test. Design, monitoring and readout each need an analyst who knows what to look for.

To be fair: open source is a fine choice for a team that already has all of this. If you have the pipelines, the targeting coverage and dedicated DS capacity, a DIY program might work.

The real comparison: total cost and time to a trustworthy answer

The question isn’t whether you can run a geo test yourself. With today’s tools, you can. The question is what it costs in spend, holdout revenue and analyst time to get an answer you’d stake next quarter’s budget on, and how long it takes to get there.

WorkMagic validates the design before launch, tells you upfront whether the test can succeed, stress-tests the readout, and puts the result to work in daily attribution and planning. That is the difference between running a test and getting an answer.

Planning a lift test? Talk to us about a design review before you spend.

Ready to improve your marketing efficiency?

Talk with a WorkMagic
growth expert