Measuring Skip-to-Swap Incrementality: Was the Add-On Really Extra Revenue?
A swap that replaces a paused shipment isn't expansion — it's substitution. Here's the holdout design that proves whether your skip-to-swap interstitial actually grew 90-day revenue per subscriber.
Quick answer
Hold out 10-20% of skip-clickers from ever seeing the swap offer, then compare 90-day revenue per subscriber between the holdout and the exposed group. If exposed RPS isn't meaningfully higher, your swap acceptance rate is measuring substitution — customers swapping in a product they'd have bought next month anyway — not incremental NRR.
Skip-to-Swap Incrementality
The portion of skip-to-swap revenue that is genuinely new — not a pull-forward or substitution of a future order.
Skip-to-swap incrementality is the additional 90-day revenue per subscriber that an exposed cohort generates versus a randomised holdout of skip-clickers who never saw the interstitial. The metric exists because swap acceptance rate flatters itself: every accepted swap looks like a saved shipment, but a meaningful share would have been purchased in a later cycle, bought as a one-off, or replaced an order the subscriber was going to resume anyway. Without a holdout, you cannot distinguish expansion from substitution, and finance ends up booking revenue that was already in the forecast.
The trap is that swap acceptance looks like a clean conversion event. A subscriber clicks Skip, sees a curated add-on, accepts it — the dashboard logs new revenue against a session that was about to log zero. It feels incremental. Mostly, it isn't.
Why swap revenue looks incremental but usually isn't
A skip request signals one thing reliably: the subscriber doesn't want this specific shipment right now. It does not signal that they don't want anything from your catalogue for the next 90 days.
When you offer a swap, three things can happen, and only one is genuinely incremental. The swap replaces the skipped box (substitution — zero lift). The swap pulls forward a purchase the subscriber would have made next cycle (timing shift — small lift, often negative on margin). Or the swap is a product they'd never have bought otherwise (true expansion — the only revenue finance should be celebrating).
The substitution tell
If your top-accepted swap SKUs overlap heavily with your top organic reorder SKUs, you're almost certainly measuring substitution. The swap is just shortcutting a purchase the subscriber would have made on the next cycle anyway.
How to detect it: the holdout design
The minimum viable test: when a subscriber clicks Skip, randomly assign them to Exposed (sees the swap interstitial) or Holdout (skip processes immediately, no offer). Keep the holdout at 10-20% for at least one full reorder cycle plus 30 days — typically 90 days for a monthly cadence.
The primary metric is 90-day revenue per assigned subscriber, not swap acceptance rate and not next-shipment retention. Assignment-based, not behaviour-based: count everyone randomised, even Exposed users who closed the modal. That's how you avoid the selection bias that makes swap dashboards lie.
Illustrative 90-day outcomes per skip-clicker, by assignment arm
| Cohort | Skip-cycle revenue | Next 2 cycles revenue | One-off purchases | Total 90-day RPS | Lift vs holdout |
|---|---|---|---|---|---|
| Holdout (no offer) | €0 | €78 | €11 | €89 | — |
| Exposed — apparel brand | €34 | €61 | €9 | €104 | +€15 (+17%) |
| Exposed — beauty refill brand | €28 | €58 | €7 | €93 | +€4 (+4%) |
| Exposed — supplements brand | €31 | €49 | €5 | €85 | −€4 (−4%) |
How to fix it: a measurement protocol that finance trusts
Run the holdout continuously, not as a one-off launch test. Subscriber behaviour shifts with season, catalogue, and price changes, so a 90-day lift measured in March tells you very little about July. A 10% perpetual holdout costs you 10% of the upside and buys you a defensible incrementality number every quarter.
Report three numbers side by side: swap acceptance rate (the vanity metric ops cares about), 90-day RPS lift (the truth metric finance cares about), and 90-day churn delta (the safety metric — see the related work on when skip-to-swap backfires into cancel-instead). When acceptance rises but RPS lift flattens, you are pulling revenue forward, not creating it.
Segment the lift, don't average it
Aggregate lift hides the SKUs and segments where the swap is purely substitutive. Break the holdout comparison down by swap category, subscriber tenure (0-3 months vs 12+), and skip reason. The categories with negative or zero lift are the ones to retire from the interstitial.
Experiment ideas worth running
Test catalogue scoping: restrict the swap menu to SKUs the subscriber has never purchased and compare 90-day RPS against a menu including reorder-frequent SKUs. The hypothesis is that novelty-only swaps trade lower acceptance for higher incrementality, and the holdout tells you whether that trade-off is worth it.
Test discount depth: most teams default to a 10-15% swap discount because acceptance rate climbs with it. With a holdout in place, you can ask the better question — does the discount eat into the incremental margin, or does it unlock genuinely new revenue? Often the undiscounted variant has lower acceptance and higher RPS lift, which is the only number that pays the bills.
Frequently asked questions
Because acceptance is self-selected. Subscribers who accept a swap are systematically different — more engaged, higher historical AOV, more catalogue-curious. Comparing them to non-accepters measures who they are, not what the interstitial did. You need random assignment between Exposed and Holdout.
For a 5% RPS lift at 80% power and 95% confidence, you typically need 8,000-12,000 skip-clickers per arm over the measurement window. At a 10% holdout split, that's about 100,000 skip events total — most subscription brands hit this in 4-8 weeks. Smaller programmes should run the holdout longer rather than shrinking the split.
For monthly subscriptions, yes — it captures the swap cycle plus two follow-on cycles, which is where substitution shows up. For quarterly cadences, extend to 180 days. The principle is one swap cycle plus at least two natural reorder opportunities, so pull-forward effects have time to surface.
It will, by exactly the size of the holdout. A 10% holdout reduces measured swap revenue by 10%. Frame it internally as the cost of knowing whether the other 90% is real. Most finance teams accept this readily once they see the alternative is booking substitution as expansion.
Yes, arguably more so. A 4% acceptance rate that's 100% incremental is worth more than a 22% acceptance rate that's 70% substitutive. Without the holdout you have no idea which programme you're running, and low-acceptance variants often get killed prematurely for the wrong reason.
It interacts directly. Some Exposed subscribers will cancel rather than swap, especially if the interstitial reads as friction — see the related guidance on when skip-to-swap backfires. Always report 90-day churn delta alongside RPS lift, because a swap programme that adds €4 RPS and 0.8 points of churn is net-negative on LTV.
Not reliably. Skip-clicker behaviour drifts with seasonality, catalogue changes, and acquisition mix, so pre/post differences usually reflect those rather than the interstitial. A perpetual randomised holdout is the only design that survives audit. Synthetic controls are acceptable only as a sanity check on top of the holdout.
A true blank — the skip processes immediately with a confirmation message. Showing the holdout a generic 'see you next month' screen is fine; showing them any product or discount contaminates the comparison. The cleanest design treats the holdout as the pre-launch baseline experience.
Randomise at the entry point — first impression of any swap surface — and measure 90-day RPS at the subscriber level regardless of how many steps they saw. Step-level analysis is useful for diagnosing the funnel but doesn't replace assignment-level incrementality.
Lead with 90-day incremental revenue per skip-event: (Exposed RPS − Holdout RPS) multiplied by skip volume. That number is auditable, ties to the P&L, and resists the substitution-as-expansion accounting that swap acceptance rate invites. Acceptance rate stays on the ops dashboard, not in the finance review.
Test ideas before you ship them
Run unlimited A/B tests, attach hypotheses to outcomes, and build a searchable archive of what works — and what doesn't.