What an Incrementality Test Costs You in Lost Sales, in Rupees

An incrementality test is not free, and pretending otherwise is why some teams avoid running one even when they badly need the answer it would give them. Here is how to think about the actual cost, with numbers.

What a test actually holds back

An incrementality test compares what sold with a channel's ads running against what would have sold without them. The common way to do that is a holdout: a share of the market — a set of regions, a customer segment, or a matched audience — sees reduced or no spend on the channel being tested for a defined period, while the rest of the market continues as normal. The campaigns are not switched off everywhere. Most of your market keeps running exactly as it was; only the held-out share is affected, and only for the test window.

Sizing the cost: a worked example

Say a brand spends ₹1 Cr a month on a channel and wants to test it. A typical holdout might pull back spend in regions representing 20% of that channel's audience for four weeks. If the channel's reported ROAS is 4x, the naive assumption is that the held-out share would have generated roughly ₹1 Cr a month × 20% × 4x = ₹80L in revenue over that month, all of which is "at risk" during the test. But that number overstates the real cost, because it assumes every rupee of reported revenue was incremental — which is exactly the question the test exists to answer. If the channel's true incremental ROAS turns out to be 1.8x once the test completes, the real revenue given up by holding back that 20% for four weeks is closer to ₹1 Cr × 20% × 1.8x ≈ ₹36L, not ₹80L. You do not know which of those numbers is closer to the truth until the test is done — which is itself the argument for running it: the gap between the naive estimate and the real one is often the size of the problem you are trying to find.

Weighing the cost against the decision

The right way to size a test's cost is against the decision riding on its answer, not against the channel's total monthly spend. A ₹30–40L cost to find out whether a ₹1 Cr/month channel is really earning 4x or 1.8x is cheap next to the alternative: continuing to scale that channel for a year on the assumption that 4x is real, when the next budget planning cycle could move tens of crores based on that number. The test is expensive in isolation and inexpensive next to what it protects you from. A test on a ₹5L/month channel, by contrast, rarely earns its cost — the decision it would inform is too small to justify even a modest four-week hit, and the channel is usually better left to the model's broader read rather than a dedicated test.

Campaigns keep running — what changes is the comparison

The part that makes teams nervous is the idea of "going dark." In practice, a test does not take the channel offline. The vast majority of the market keeps seeing the same campaigns at the same spend; only the held-out slice is adjusted, and the point of the exercise is the comparison between the two groups, not a company-wide pause. Framed that way, the real question is not "can we afford to stop advertising for a month" — the answer is you are not stopping — but "can we afford to not know, for one more budget cycle, whether this channel's real number is 4x or 1.8x."

When a test is not worth running

Not every channel justifies a test, and running one everywhere, every quarter, is its own kind of waste. A test earns its cost when three things are true: the channel takes a meaningful share of the budget, the reported number and the model's estimate disagree by enough to matter, and a real decision — scale it, cut it, hold it — is waiting on the answer. If all three are true, the ₹30–40L question above is worth asking. If a channel is small, if the model and the platform already roughly agree, or if no decision is actually pending, the test's cost buys you confirmation of something you already knew, which is the one thing not worth the price.