Bayesian A/B testing isn't just a thought experiment — it's genuinely run in production at real companies, and recent work backs up the same conclusion worked through here a few weeks ago: shipping whenever the posterior expected value clears a lenient bar just above zero maximizes the payoff across every experiment you run. What that literature quietly doesn't have is anything resembling goal metrics and guardrail metrics — the vocabulary this whole blog has been built around. Going looking for where that split actually lives turned into a longer, messier question than I expected.
Confirmed in Theory, Missing in Practice
The recent papers I've been reading confirm the simple threshold: launch whenever the posterior expected value is positive, which in practice means a chance-to-win only just above 50%, not a strict 95%-style bar. That part lines up with what this blog already landed on. But almost none of that literature — and almost none of the big-tech writeups built on top of it — talks about goal metrics and guardrail metrics as a pair. The exceptions are notable precisely because they're exceptions: Spotify's Confidence platform treats this distinction in real depth, Statsig — built out of a team with a Meta pedigree — ships a statistics engine built around the same concept, and Airbnb is known to run its own guardrail metrics as well. But across the wider literature, goal and guardrail metrics are almost entirely a frequentist-flavored idea, even at companies that otherwise lean Bayesian everywhere else.
The Frequentist Guardrail Rule Might Be Too Lenient, Not Too Strict
The standard practice, wherever guardrails do show up, is frequentist almost by default: maximize the goal metric, unless the guardrail metric shows a significant decline. But failing to find a significant decline in something like company-wide revenue is not the same as there being no decline. Plenty of real declines sit in the not-quite-significant range — a p-value of 0.15 or 0.20 is common enough — and under the standard rule, those cases pass through untouched. If you blocked launches in exactly that ambiguous range instead of letting them through, realized profit should go up, not down. It's the mirror image of the earlier point about requiring proof before shipping: here, requiring proof before rolling back is what's letting real, if modest, harm through.
What Even Is a Guardrail, If the Goal Is Also "Don't Go Down"?
There's a more basic conceptual snag underneath all of this. A guardrail exists so a metric doesn't get worse. But if what you actually want is to maximize revenue, isn't revenue just the goal metric at that point, not a guardrail? The recent Bayesian papers are entirely about maximizing a goal metric — which leaves an open question about why goal and guardrail metrics exist as two separate things at all, if the guardrail is something you'd also want to push up given the chance.
The resolution, as far as I can tell, is really about scope. If a company-wide guardrail doesn't move, but clicks or favorites go up on one specific placement inside a single squad's surface, that's a legitimate signal — a genuinely local goal metric improving without disturbing anything broader. The natural long-term proxy is usually just short-term revenue itself. And if a policy's actual surface is narrow — one placement, one squad — then of course the company-wide guardrail barely moves; it was never going to. What should move is the local revenue on exactly the surface the policy touched. The guardrail and the goal metric aren't really answering the same question — one is scoped to the whole company, the other to wherever the experiment actually acts.
Two Guardrails Is Manageable. Two Guardrails Plus a Goal Metric Isn't.
With a single guardrail, the shape of the problem is simple: maximize the goal metric, subject to the guardrail not dropping. Add a second guardrail, and requiring both to individually clear "posterior expected value positive" stops being jointly optimal — the same AND-condition problem that showed up with two goal metrics earlier. At that point you need something more like a C-level utility function that actually specifies how the two guardrails should combine — maybe a break-even floor, maybe maximizing transaction volume outright, maybe holding seller-side ROAS above some minimum.
Two guardrails alone is at least tractable. Two guardrails plus a goal metric on top is where it stops being a clean threshold problem. In a fully Bayesian frame, that probably means defining something closer to a company-level welfare function — or at minimum a squad-specific objective function — where the guardrails act as budget constraints and the goal metric is the thing actually being maximized. Secondary metrics complicate this further, but I think they're a separate question: their role usually isn't optimization at all, it's explanation — confirming the result is moving for the reason you expected, not changing what you'd decide. That's a different lens entirely, and probably its own piece later rather than something to fold in here.
An Open Thought to Close On
The more I sit with this, the more it seems like frequentist statistics was never really built to answer "what should we maximize" in the first place — it's built to produce a theory that's correct, a guarantee that holds, regardless of what the business objective actually is. Bayesian methods, by contrast, slot naturally into an optimization frame: maximize expected value, subject to constraints. Which framework actually belongs underneath a goal-metric-plus-guardrail-constraint structure is a genuinely open question for me right now, not a settled one — and probably the right next thing to dig into.