Why on-and-off lift measurement tests cost more than you think

Periodic incrementality testing has a hidden price tag. Every time you pause a campaign for a blackout-style lift test, you're paying for algorithm re-learning, missed seasonal windows, and results that often arrive after the decision they were meant to inform. For most mobile app advertisers, that cost could be higher than the cost of the test itself. This is why always-on incrementality measurement is the new industry standard.

‍

What is a periodic (on-and-off) lift test?

A periodic lift test is a time-boxed incrementality experiment. The most common version splits users into a Treatment group (sees ads) and a Control group (doesn't), lets the test run for a set window then pauses it, calculates the result, and moves on until the next test.

A more disruptive variant is the blackout test, which pauses the entire campaign for the test window, then compares performance before and after the blackout. Where a Treatment/Control split only holds back a portion of the audience, a blackout holds back everyone, making its ramp-up and opportunity cost even steeper.

Either way, the test answers one question, for one moment in time: did this campaign cause conversions that wouldn't have happened otherwise? The problem is what happens in between tests, and what the pause itself costs.

‍

Why periodic lift testing costs more than it looks like

‍

1. Campaigns lose their ramp-up when they pause

Programmatic buying algorithms learn from continuous signals. When a campaign pauses for a test (or pauses because the test requires withholding ads from part of the audience), the algorithm resets. Once the test ends and the campaign resumes at full volume, it re-enters a ramp-up phase, rebuilding performance it had already achieved before the pause. That new ramp-up isn't free. It shows up as depressed performance for days or weeks after every single test cycle.

‍

2. Snapshot results miss the context that actually drives performance

A periodic test only captures a snapshot of whatever window it happens to run in. If that window overlaps with a seasonal peak, a promotion, a competitor's campaign push, a special date, or any other market shift, you're not measuring your campaign's actual incremental value, but rather the incremental nature of that specific moment.

It doesn't mean the results are wrong, just incomplete. Static snapshots can't separate your campaign's baseline incrementality from the noise happening around it.

‍

3. Results arrive after the moment they were meant to inform

By the time a periodic test is designed, run, and analyzed, weeks have usually passed. Any budget decision the test was meant to support (e.g., cutting an underperforming channel, doubling down on a strong one, adjusting for a seasonal shift) has often already been made on gut instinct, or missed entirely.

Always-on vs. periodic lift measurement: side-by-side comparison

Not all lift tests are equal

Even among periodic and always-on approaches, the underlying test methodology changes what you're actually measuring. The three most common designs:

Ghost Bids (the methodology behind Jampp's Always-on Lift Measurement) solves the two problems that make most incrementality testing either expensive (Placebo/PSA) or noisy (ITT): it never serves an extra ad, and it never counts a user who wasn't actually eligible to be exposed.

‍

How always-on lift tests stay accurate without ever pausing

Jampp's Always-on Lift Measurement stays statistically valid every single day by using a rolling window: a fixed number of days (long enough to reach statistical significance) that moves forward daily, incorporating the newest day of data and dropping the oldest, so the sample size needed for a reliable result stays constant.

The result: lift is recalculated daily, using a window that's always current, never diluted by data that's weeks old, and never waiting for a test to "finish." At Jampp, we calculate a rolling window for each campaign individually, since every campaign reaches its needed sample size at a different pace.

‍

Case studies: always-on lift measurement in action

‍

FREENOW

The European ride-hailing app, leveraged Jampp's Always-on Lift Measurement to validate that its campaigns were generating net-new rides, not just capturing riders who would have converted anyway. Result: an 86% increase in new riders, 122% increase in rides, and 30% campaign lift — measured continuously, not in a single test window.

‍

Wallapop

The Spanish marketplace app used always-on lift to separate genuine new listers from users who would have listed regardless. Result: 93% growth in listers and a 92% lift in conversions.

‍

More FAQs: On-and-off vs. always-on lift measurement

Does pausing a campaign for a lift test really disrupt performance? Yes. Delivery algorithms rely on continuous learning signals. A pause (full or partial) resets that learned state, forcing a ramp-up period once the campaign resumes.

Is always-on measurement more expensive than a periodic test? No. Methodologies like Ghost Bids run continuously at no added media cost, since they don't require serving placebo ads or reserving a separate test budget. The only tradeoff is a small slice of traffic, typically around 6%, set aside for the control group.

How much of my traffic needs to be held back for always-on testing? When working with Ghost Bids, we recommend having around 6% of users randomly assigned to the control group—small enough to preserve statistical power without meaningfully reducing campaign reach. That assignment is based on a random split of user IDs to guarantee that the control group isn’t skewed toward any particular type of user. 

Can I still run periodic tests alongside always-on measurement? You can, but for most advertisers it's redundant. Always-on measurement already answers the question periodic tests are designed to answer continuously, and without the disruption.

‍

Start measuring the incremental impact of your campaigns today

Want to see what continuous incrementality measurement looks like on your own campaigns? Request a demo of Jampp's Always-on Lift Measurement.

‍