Startup Finance Guide
Sales & Fintech Tools

Guides on CRM deal detection, debt collection voice AI, FDCPA compliance, and fintech automation. Multi-vendor comparisons with compliance scores, implementation timelines, and honest limitations.

How to Measure Whether a Customer Nudge Actually Changed Behavior, Not Just Got Opened

How to Measure Whether a Customer Nudge Actually Changed Behavior, Not Just Got Opened

Opens and clicks measure who was already engaged, not what the nudge caused. Compare a randomized holdout against everyone assigned to the nudge, size the test for the smallest lift worth acting on (about 2,000 users per group for a 4 point lift on a 27% baseline), count outcomes from assignment, and recheck at 30 to 90 days. Userlens Lumi tracks adoption behavior but its site describes no built-in holdout; PostHog, Amplitude, Statsig, GrowthBook, Appcues and Pendo document control or holdout features, several only on paid tiers.

MABy Mashkoor AlamEditor, Startup Finance Guide
Reviewed by Startup Finance Guide Editorial Team

This article is for informational purposes only and does not constitute financial, tax, or legal advice. Consult a qualified professional for guidance specific to your situation.

Reviewed for financial accuracy by the Startup Finance Guide editorial team. Our editors cross-reference all claims against platform documentation, pricing pages, and primary regulatory sources. Last reviewed: November 16, 2026.

To find out whether a customer nudge changed behavior, randomly withhold it from part of the eligible audience, then compare how often each group completes one target action that you wrote down before launch. Count completions over a fixed window from assignment. The gap between the two groups, in percentage points with a confidence interval, is the nudge's effect. Opens and clicks cannot give you that number, because the people who open are already the more engaged users. Userlens's adoption agent, Lumi, is built around this idea: its site says success is "measured in product behavior, not opens or clicks." Its site does not describe a built-in control group, though, so you still need a random holdout from Lumi itself or from an experiment tool such as PostHog, Amplitude, Statsig or GrowthBook. In-app guide tools Appcues and Pendo now run no-guide control groups natively. For a sense of scale, a 4 point lift on a 27% baseline needs about 2,000 users in each group to detect reliably, and the average nudge in 126 government trials moved behavior by 1.4 points.

Tools that can send or measure a nudge

The table only lists control and holdout features where the vendor's own documentation describes them, checked on 16 Nov 2026.

ToolWhat it does for nudgesControl or holdout, per its own docsPlan needed for thatWatch out for
Userlens (Lumi)Writes and sends personal email or Slack nudges from product and database data, then tracks whether adoption movedNot described on its siteFree plan: 100 profiles, 300 emails a month, 1 campaignLumi may skip qualifying users, so compare the whole eligible pool, and add your own random holdout
PostHogFeature flags, experiments and product analytics in one placeExperiments with control variants; holdout groups (1% to 10% recommended)Free monthly allowance of 1M flag requests; experiments billed with flagsHoldout users are left out of experiment results, so you analyze them separately in insights
AmplitudeExperiment holdouts, plus A/B tests on in-app guides and surveysHoldout groups in Amplitude Experiment (1% to 10% recommended); a no-guide control in Guides and Surveys experimentsHoldout groups and Guides and Surveys A/B testing are each an add-on on Growth and Enterprise; not on Free or PlusExposure only fires when a user actually sees the guide, so read results by assignment
Statsig (acquired by OpenAI in 2025; brand and customers taken on by Amplitude in May 2026)Experimentation and feature gatesHoldouts of 1% to 50%, typically 1% to 2%Holdouts need Pro, $150 a month; CUPED and sequential testing are listed on every tier, including the free Developer tierBuilt for long cumulative holdouts across many launches, not single campaigns
GrowthBookWarehouse-native experiment analysis, open-source coreHoldouts (default 5%)Enterprise; CUPED and sequential testing need Pro at $40 a seat a monthThe free Starter tier has no holdouts, and holdout metrics cannot use conversion windows
AppcuesIn-app flows with goalsControl experiments: a blocked control group, 50/50 by defaultIts docs do not state the planGoals count only within 7 days of group assignment
PendoIn-app guidesA/B tests with an optional "No guide" groupGuides ProAttribution window is capped at 14 days

If you are still choosing a sending tool, our roundup of product adoption tools for B2B SaaS compares them on more than measurement.

Why opens and clicks cannot answer the question

An open tells you a message reached someone. It says nothing about what that person would have done without it. The users who open nudges are disproportionately the ones already logging in, already curious and already likely to try the feature. Comparing openers with non-openers measures that difference in people, not the nudge.

A hypothetical shows how large the error can be. You nudge 5,000 users and 2,000 open. Openers adopt the feature at 44.5% and non-openers at 22%, so the "openers versus non-openers" gap is 22.5 points. But the whole nudged group adopts at 31%, and a randomly chosen holdout that got nothing adopts at 27%. The real effect is 4 points. The other 18.5 points were selection.

So compare groups by assignment, not engagement. Everyone you randomly assigned to receive the nudge stays in the nudged group, whether or not they opened it. Statisticians call this intention to treat. If you also want the effect on people who actually saw the message, divide the intention-to-treat effect by the share who saw it. In the example, 4 points divided by a 40% open rate gives about 10 points among openers. That calculation assumes the nudge did nothing for people who never opened it and that opens are recorded accurately. Subject lines and previews can reach people who never open, so treat the result as an approximation.

The method in eight steps

1. Pre-register the target behavior

Write down, before anything ships, the one action that would show the nudge worked. Pick a completed action that carries value, such as finishing a feature setup, scheduling a report or accepting an invite. If one-off trial use is a risk, add a persistence rule, for example "used the feature on at least two separate days within 14 days of assignment."

Your plan should name five things: the primary metric, the eligible audience, the analysis window, the guardrail metrics (more on those in step 8), and the smallest lift worth acting on. Date it and share it. Academic researchers file this kind of plan on registries such as the OSF preregistration service; a dated document in your team wiki does the same job. Once results arrive, the plan stops anyone from switching to a friendlier metric or a flattering segment.

2. Fix the eligible audience and its baseline

Define eligibility with traits that exist before the nudge, such as "active in the last 14 days, on a paid plan, has never used bulk export." Then pull the target action's rate for that same definition over at least two previous cycles. If it already swings from 24% to 29% month to month, a single before-and-after comparison is worthless, because ordinary noise is bigger than most nudge effects.

Use product data for the baseline, not email statistics. Userlens says Lumi combines a product database (Postgres, Snowflake or BigQuery) with behavioral analytics from PostHog, Mixpanel or Amplitude; whichever tool you use, the baseline and the outcome should come from the same event source. Our guide to product analytics platforms for CS teams covers tools that make this pull easier for non-engineers.

3. Randomize, at the right level

Assign each eligible user to "nudge" or "holdout" by chance before anything is sent, and keep the holdout clear of every version of the campaign. Hash-based assignment from a feature flag (what PostHog, Amplitude, Statsig and GrowthBook do) gives the same user the same group every time.

Three details decide whether the split is clean:

  • Randomize by account when colleagues share a workspace. A nudged admin can tell a teammate in the holdout, which shrinks the measured effect. Account-level randomization costs power, though. With accounts of about eight users and a modest correlation between colleagues (an intraclass correlation of 0.05), you need roughly 1.35 times as many users, so a 2,000-per-group test becomes about 2,700 per group.
  • Use a 50/50 split for a single campaign test. The small 1% to 10% holdouts that PostHog and Statsig recommend are for long-running holdouts that sit outside many experiments at once. For a one-off nudge, a small control group is expensive: detecting a 4 point lift from 27% needs about 4,000 users in total at a 50/50 split, about 6,100 at 80/20, and about 10,800 at 90/10.
  • Check the split. If you planned 50/50 and got 52/48 on 10,000 users, something in assignment or logging is broken. Fix that before reading any outcome.

4. Size the test: minimum detectable effect and sample size

The minimum detectable effect is the smallest lift your test can reliably spot. Set it from the business case ("anything under 2 points is not worth the campaign"), then compute the sample. At 95% confidence and 80% power, with a 27% baseline:

Lift you want to detectUsers needed in each group
1 point (27% to 28%)about 31,300
2 points (27% to 29%)about 7,900
4 points (27% to 31%)about 2,000
6 points (27% to 33%)about 900

These come from the standard two-proportion formula, the same one behind Evan Miller's sample size calculator. Halving the effect you want to detect roughly quadruples the sample. Appcues builds this step into its control experiments: you enter a baseline conversion rate and a minimum detectable effect, and it reports a minimum sample size and estimated duration.

Many early-stage B2B products cannot reach 7,900 users per group for one campaign. Your options are to accept a larger minimum effect, run the test longer, pool several similar campaigns under one holdout, or use variance reduction. CUPED, from a 2013 Microsoft paper, uses each user's pre-period behavior to cut noise. PostHog supports it on mean metrics and ordered funnels, and GrowthBook includes it on its Pro plan.

5. Set the attribution window from assignment

Count the outcome over a fixed window that starts when a user is assigned, not when they open. The holdout never receives anything to open, so a window anchored on opens would compare different time spans. Match the window to how long the action takes. Scheduling a report might happen within a day; rolling a feature out to a team can take three weeks.

Vendor defaults are often shorter than B2B behavior warrants. Appcues counts goals only within 7 days of group assignment, and Pendo tracks conversions for at most 14 days after the first guide view, a window anchored on viewing rather than assignment. If your action takes longer, export the assignment list and compute the outcome in your own analytics. Also exclude anyone who had already completed the action before assignment, which Appcues does automatically.

6. Calculate the lift and its confidence interval

When the window closes, subtract the holdout's rate from the nudged group's rate. Then put an interval around it, because the interval is what tells you whether the lift is real.

Here is a worked example. Nudged group: 5,000 users, 31% adopted (1,550). Holdout: 5,000 users, 27% adopted (1,350). The lift is 4.0 points, or 200 extra adopters. The standard error of the difference is about 0.91 points, so the 95% interval runs from about 2.2 to 5.8 points. It stays above zero, so the nudge worked.

Now keep the same rates but give the holdout only 556 users (a 10% holdout). The interval widens to roughly 0.1 to 7.9 points. The best estimate is unchanged, but you can barely rule out zero. With 500 in the holdout, the interval crosses zero. Small holdouts produce point estimates with little precision.

Report the absolute lift first and the relative lift second. In a field experiment with more than 53,000 college students, Susan Athey and colleagues found that nudging raised early financial-aid filing by 6.4 percentage points over a 37% baseline. That is a 17% relative gain, but in plain terms about one extra student in 16 acted, which is the number a budget owner needs.

When you genuinely cannot randomize, for example because a nudge went to every customer on one plan, difference-in-differences is the next best design. Compare the before-and-after change in the nudged group with the change in a similar group that was not nudged yet. If early recipients went from 26% to 34% adoption (up 8 points) while the not-yet-nudged group went from 27% to 29% (up 2 points), the estimate is 6 points. It rests on an assumption you cannot fully test: that both groups would have moved in parallel without the nudge. Plot several pre-periods to check that their trends matched before you trust it. Scott Cunningham's chapter on difference-in-differences explains the failure modes.

7. Separate novelty from lasting change

Many nudges produce a spike that fades. Users try the feature once because it is new and then go back to old habits. Measure the same outcome for both groups again at 30, 60 and 90 days after assignment, using the persistence rule from step 1. If the nudged group's advantage shrinks toward zero, the campaign bought trial, not adoption.

For a cumulative view, keep a small long-running holdout that receives none of your nudges for a quarter or more. This is what PostHog, Statsig and GrowthBook holdouts are designed for. PostHog describes holdouts as useful for effects "typically over a few months," and Statsig recommends running one for three to six months before releasing it. Repeated messages can also lose their effect. Track the lift from the first, second and third message separately, and stop sending when the marginal message stops adding anything.

This is also where nudge measurement connects to retention work. Our guide to identifying at-risk customers before they churn covers the signals worth tracking as longer-term outcomes.

8. Watch guardrails and set a stopping rule in advance

A nudge can win on its target metric and lose elsewhere. Pick two or three guardrail metrics before launch and compare them between the groups with the same method:

  • Displacement: weekly active use of the core product, or of the feature the user would otherwise have used. A drop means attention was moved, not added.
  • Fatigue: unsubscribes, Slack mutes, support tickets and complaints in the nudged group.
  • Revenue risk: downgrades and cancellations over a longer window, since a rushed decision may surface weeks later.

Decide the stopping rule at the same time. Checking results every day and stopping the first time they look significant inflates false positives, because random noise crosses the line often in small early samples. Appcues's own documentation warns against stopping "the first time results cross the significance threshold." Either fix the duration in advance or use a sequential method built for repeated looks (GrowthBook offers sequential testing on Pro). A guardrail can still trigger an early stop, for example "pause if unsubscribes in the nudged group exceed the holdout's by more than 1 point."

What results to expect

Small effects are normal, and big early wins usually shrink. Stefano DellaVigna and Elizabeth Linos assembled 126 randomized trials from two US government nudge units, covering more than 23 million people. The average effect was 1.4 percentage points, an 8.1% increase over control. Nudge trials published in academic journals averaged 8.7 points. The authors found that publication bias, made worse by low statistical power, could account for the whole gap.

A 2025 second-order meta-analysis in the Journal of Behavioral Decision Making pooled 14 meta-analyses covering 1,638 primary studies and about 30 million participants. The average effect was small (d = 0.27) and fell to near zero (d = 0.004) after adjusting for publication bias, and most of the underlying meta-analyses were rated low or critically low in quality.

Effects also differ by who receives the nudge. In the Athey study, targeting half of the students with the authors' preferred policy captured about 75% of the benefit of nudging everyone, while targeting the students least likely to file on their own was counterproductive. Before you build a targeting model, compare the pre-registered segments from step 1. Treat any segment result you did not plan for as a hypothesis for the next test, because slicing ten ways makes one false positive likely.

Log every nudge test, including the nulls. If 2 of 10 campaigns deliver a lasting lift with clean guardrails, 20% is your real hit rate.

Tool by tool

1. Userlens (Lumi)

Userlens describes itself as "a customer adoption and retention engine" for product-led and B2B SaaS teams, and Lumi is its adoption agent, launched on September 1, 2026. According to Userlens's site, a team sets a campaign brief once (goal, audience, tone, exclusions and limits), and Lumi writes and sends personal email or Slack nudges based on each person's product usage. It enforces frequency caps, quiet hours, never-contact flags and skip-if-unsure rules before sending, and a scoreboard shows "whether the intended adoption behavior moved." It connects to databases such as Postgres, Snowflake and BigQuery and to PostHog, Mixpanel and Amplitude.

The launch post says Lumi "measures whether behavior changed, not merely whether someone opened or clicked." The pricing page lists a free plan (100 profiles, 300 personalized emails a month, one active campaign, PostHog and database integrations), Starter at $100 a month for 10,000 profiles, and Growth from $1,000 a month billed annually.

Watch out for: the site does not describe a randomized control or holdout group, so a scoreboard showing adoption rising among nudged users is not, on its own, proof that the nudge caused it. Because Lumi can decide to wait or skip a qualifying user, the people who got messages are not a random subset either. Assign a random holdout yourself (Lumi's contact rules let you define who it must leave alone, or you can run the split through a feature flag) and compare the whole eligible pool against it. Lumi sends by email and Slack, not in-app guides. It is also new, so there is little public track record yet. Our explainer on driving adoption of features customers already pay for covers the campaign side.

2. PostHog

PostHog combines product analytics, feature flags and experiments, with both Bayesian and frequentist statistics and CUPED variance reduction. Its holdout groups exclude a percentage of users from specific experiments or all of them. Users are assigned by hashing the user ID, and PostHog recommends 1% to 10%. The pricing page lists a free monthly allowance of 1M feature flag requests, with experiments billed with flags.

Watch out for: holdout users are not included in experiment results. You compare them with other users in insights by breaking down on the holdout flag value. Editing a holdout's percentage while experiments use it changes who is excluded mid-test, which PostHog itself warns against.

3. Amplitude

Amplitude covers both halves. Amplitude Experiment holdout groups keep a share of users out of experiments to measure long-term effects. For in-app nudges, Guides and Surveys experiments run A/B or multi-armed bandit tests in which "users in the control group don't see the guide or survey."

Watch out for: holdout groups and Guides and Surveys A/B testing are each an add-on on the Growth and Enterprise plans, so neither is in Free or Plus. Amplitude's docs note that the exposure event fires only when a user actually sees the treatment, so if another guide blocks it, no exposure is logged. Analyze by assignment where you can.

4. Statsig

Statsig is an experimentation and feature-gating platform. OpenAI agreed to acquire it in September 2025. On May 5, 2026, Amplitude announced that it would take on Statsig's brand and customers and keep developing the platform. Amplitude calls this a strategic partnership, while a note added to Statsig's blog describes it as an acquisition by Amplitude. The original Statsig team stayed at OpenAI. Its holdouts withhold "typically 1% to 2%" of users from a set of features (the console allows 1% to 50%) and compare them against an equal-sized random sample of everyone else.

Watch out for: holdouts are a Pro feature at $150 a month on the pricing page, not part of the free Developer tier, although the same page lists CUPED and sequential testing on every tier. Holdouts are designed for the combined effect of many launches over months, so for a single nudge campaign a regular 50/50 experiment is the better tool. Statsig changed hands twice in eight months and its original team stayed at OpenAI, so check the roadmap before committing.

5. GrowthBook

GrowthBook runs experiment analysis on top of your own warehouse, and most of its code is MIT-licensed and self-hostable, with enterprise features under a separate license. Its holdouts default to 5% of traffic and measure the cumulative impact of features against a same-sized sample of everyone else.

Watch out for: holdouts are an Enterprise feature, and CUPED and sequential testing need Pro at $40 a seat a month, according to its pricing page. The free Starter plan is limited to three users and one project. GrowthBook also blocks metrics with conversion windows in holdouts, so long-run holdout metrics need lookback windows instead.

6. Appcues

Appcues control experiments block a random control group from seeing a flow and compare goal conversion between the groups, defaulting to a 50/50 split. They report lift and a 95% confidence measure only after the minimum sample size is reached.

Watch out for: the flow must be published, set to show once and have a goal attached, and conversions count only within 7 days of assignment. Appcues's documentation does not say which plans include the feature.

7. Pendo

Pendo's A/B testing with guides compares two guides, or one guide against a "No guide" group, on a single conversion event, with experiments running up to three months. It flags results with 95% or higher confidence as significant.

Watch out for: it requires Guides Pro, and the attribution window is capped at 14 days. Pendo's own article notes that significance "doesn't indicate the magnitude of the outcome," so report the lift and its interval, not only the badge. If you onboard at scale with in-app tours, our piece on onboarding SaaS customers at scale covers where guides fit.

Turn the result into a decision

Once you have a lift with an interval, convert it into cost per extra adopter. Suppose a campaign reaches 10,000 nudged users and the holdout also has 10,000. The nudged group adopts at 29% and the holdout at 27%, a 2 point lift, or 200 extra adopters. If the campaign costs $1,000 in total, each extra adopter costs $5.

Run the same arithmetic on the interval. The 95% interval here is roughly 0.8 to 3.2 points, or about 80 to 320 extra adopters, so the cost per adopter is about $3 to $13. If a user who adopts the feature is worth $200 in retained revenue, the campaign pays at either end. If a user who adopts the feature is worth $10, you need a bigger test before scaling. Write the result, the interval and the cost into the test log, and judge the next nudge by the same rule.

Frequently asked questions

How long should a nudge experiment run?

Run it until each group reaches the sample size you calculated in advance, and for at least one or two full weekly cycles so weekday patterns even out. The outcome window should then cover the time users need to complete the action, measured from assignment. Do not stop early because results look good; that inflates false positives unless you are using a sequential method designed for repeated checks.

How many users do I need to measure a nudge?

It depends on your baseline rate and the smallest lift worth detecting. At a 27% baseline, 95% confidence and 80% power, detecting a 4 point lift needs about 2,000 users per group, a 2 point lift about 7,900, and a 1 point lift about 31,300. Randomizing by account rather than by user raises those numbers. Small B2B audiences can pool similar campaigns, accept a larger minimum effect or use CUPED to cut noise.

Can I measure a nudge without a holdout group?

Yes, with weaker evidence. A staggered rollout lets you compare early recipients with users who have not been nudged yet using difference-in-differences, which removes shared trends such as seasonality. It depends on both groups following parallel trends without the nudge, so check several pre-periods first. A simple before-and-after comparison with no comparison group is not reliable for effects of a few points.

Are open and click rates useful at all?

Yes, as diagnostics, not as results. Low opens point to timing or subject lines; strong clicks with weak completion point to friction inside the feature. They also let you convert the intention-to-treat effect into an effect among people who saw the nudge. The verdict on whether the nudge worked should come from the holdout comparison.

How we put this together

We checked every product claim against each vendor's own documentation and pricing pages on 16 Nov 2026, including Userlens, PostHog, Amplitude, Statsig, GrowthBook, Appcues and Pendo, and listed control or holdout features only where those docs describe them. Research figures come from the original papers on arXiv, NBER and the Journal of Behavioral Decision Making. We recomputed every worked example (sample sizes, confidence intervals, difference-in-differences and cost per adopter) with the standard two-proportion formulas.


This article reflects publicly available information as of November 2026 and does not endorse any specific platform. Vendor capabilities, plan gates and pricing in this category change frequently. Verify current terms directly with each provider.

Last verified: 2026-11-16