Every experiment carries two different kinds of metrics, even when nobody labels them that way. There's the metric you're trying to move — sign-ups, checkout conversion, watch time — and there are the metrics you're trying to protect while you move it — revenue, latency, unsubscribe rate. Almost every experimentation platform runs both through the exact same statistical test. That's the mistake this article is about.
Two Metrics, Two Different Questions
Call the first kind a goal metric: the thing the experiment exists to improve. The question you're asking of a goal metric is "did this get better?" You want evidence of a difference, specifically in a good direction, and you're happy to conclude nothing happened if the data doesn't clearly show an improvement.
Call the second kind a guardrail metric: the thing the experiment must not quietly break. The question you're asking of a guardrail is completely different — not "did this change?" but "did this stay okay?" You're not hoping to detect a difference. You're hoping to rule one out.
Those are not the same question, and treating them as if they were is where most experimentation programs quietly go wrong. A test built to answer "did this change?" does not automatically answer "did this stay okay?" — even though it's tempting to assume it does, because it's the only test most teams have in the toolbox.
The Test Everyone Reaches For
The standard two-sample test used for goal metrics sets up a null hypothesis of "no difference" and an alternative of "there is a difference":
This is a superiority test. It's built to detect a difference, and it is very good at that job. When the p-value crosses your threshold, you have real evidence that treatment and control aren't the same. That's exactly what you want when you're evaluating a goal metric.
The trouble starts when the same test gets pointed at a guardrail metric — revenue, say — with the same null hypothesis of "no difference." A non-significant p-value on that test tells you that you failed to find evidence of a difference. It does not tell you that no difference exists. Those are not the same claim, and the gap between them is exactly where guardrail failures hide.
Why "Failed to Reject" Isn't "Proved Safe"
Take a typical result: in a real A/B test, revenue per user dropped 0.6%, with a p-value of 0.32. This test fails to reject the hypothesis of no difference between treatment and control. But in reality, it's more likely that this experiment caused a decrease than not. There isn't enough confidence here to say the treatment definitely caused real harm — but some amount of decline is still the more likely outcome, not the less likely one.
What the business actually wants from a guardrail check, though, is to show that revenue per user didn't decrease. The standard superiority test isn't built for that job — it's built to prove a decrease occurred, not to prove one didn't. Confirming a guardrail is safe calls for a different test altogether: a non-inferiority test, built specifically to prove revenue didn't decrease, rather than to prove that it did.
A Concrete Version of the Problem
Suppose a checkout redesign is being tested. The goal metric — conversion rate — was carefully powered: the team ran a sample size calculation, decided they needed to detect a 2 percentage point lift, and estimated two weeks of traffic would be enough to do that reliably.
Revenue per user rides along as a guardrail on the same sample. Nobody ran a separate power calculation for it — it was just treated as something to monitor, on the assumption that "not significant" would mean "fine." At the end of the experiment, revenue per user is down 0.4% with a p-value of 0.32. Not significant. The team reads this as "no significant impact on revenue" and ships.
But revenue is a noisier, higher-variance metric than conversion rate, and the sample was sized for conversion — not for revenue. Whether the guardrail actually needed a larger sample than that to be trustworthy, or didn't, isn't something you can just assume either way. It's something you'd have to actually calculate.
Why Guardrails Are Systematically Underpowered
This isn't a one-off mistake — it's the default outcome of how experiments get designed. Sample size is almost always calculated for the goal metric alone, using the minimum detectable effect the team actually cares about for that metric. Guardrail metrics are added afterward, evaluated on whatever sample the goal metric happened to need, with no separate power analysis of their own.
- Guardrails are often higher-variance than the goal metric — revenue and latency both tend to have fatter tails than a binary conversion event, which inflates the standard error for a given sample size.
- Guardrails are rarely powered on purpose. Nobody asks "how small a revenue drop do we need to be able to detect?" before launch — the guardrail just inherits whatever sample the goal metric was sized for.
- A wide, underpowered interval will contain zero almost by default, regardless of whether a real (and meaningful) regression is happening underneath it.
Put those three together, and "non-significant guardrail" stops meaning "guardrail confirmed safe" and starts meaning "we didn't collect enough evidence to know either way" — which is a very different thing to write in a launch decision doc.
What a Guardrail Test Actually Needs
If a superiority test can't answer "did this stay okay?", something else has to. That something else needs to flip the null hypothesis around: instead of assuming "no difference" and looking for evidence against it, it should assume "the treatment is meaningfully worse" and require the data to actively rule that out. Only rejecting that null gives you the kind of evidence a guardrail decision actually needs — a positive confirmation of safety, not a failure to find harm.
That flipped-null approach has a name — non-inferiority testing — and it's built specifically to answer the guardrail question instead of the goal-metric question. The next article in this series looks at what actually happens, quietly, when teams don't draw that distinction — starting with the loss created when the standard test comes back non-significant but negative.
What Comes Next
Before getting into the mechanics of non-inferiority testing, it's worth sitting with just how common — and how invisible — the "flat means fine" mistake is. The next article walks through exactly how a genuinely harmful change survives a guardrail check and ships anyway, using nothing more exotic than a small sample and an honest reading of a p-value.