Contents
Retry budgets are the standard answer to retry storms: cap retries at each hop to some percentage of traffic, and the theory is that a struggling downstream service never gets hit with more than a bounded multiple of its normal load. Uber’s engineering team published the math showing why that theory is wrong for any call chain with real depth, and shipped a fix that’s been running across their service mesh: a header-based protocol that tells each service whether an error it’s looking at is actually its own to retry, or someone else’s.
The formula everyone gets wrong
Take a straight call chain, node A through node G, seven hops deep, where A receives Ƞ requests per second and every node passes the request straight through in the steady state. Now node D starts failing, and every node in the chain retries once on failure (one regular attempt, one retry). The number of requests each node ends up serving isn’t proportional to the failure, it compounds: with R retries at each hop, a node at depth ɗ serves R^ɗ × Ƞ requests. At depth 3, one retry per hop means 2³ = 8x the baseline load hitting node D and everything below it.
Retry budgets, capping retries to some percentage B of traffic per hop, look like they fix this: the formula becomes (1+B)^ɗ × Ƞ instead of R^ɗ × Ƞ. A 10% budget at every hop gives you 1.1×, 1.21×, 1.33× as depth increases, then flattens once retries exhaust the budget. That looks bounded and safe. It is bounded. But it’s bounded at the wrong scope. A 10% budget doesn’t mean “10% more load on the failing service.” It means every hop between the caller and the failure adds its own 10%, compounding down the chain. At depth 6 you’re at 1.33x, which sounds fine until you remember that number is being added on top of whatever load node D is already failing to handle.
Uber’s fix restricts the retry to the single edge closest to the failure. In the same seven-node example, if only the C-to-D edge is allowed to retry, and A through C are prevented from retrying on that same error, the whole downstream chain from D to G sees a flat 1.1x instead of a compounding 1.33x. The math is straightforward once you see it, but it requires something retry budgets alone can’t give you: knowing, at each hop, whether the error in front of you originated here or was just passed through.
Why retrying can make things worse, not better
There’s a second, less intuitive result in the same post. Retrying only helps the caller’s perceived availability when the callee’s errors are independent events, the kind where trying again has a real chance of hitting a healthy instance. Uber ran the numbers across a range of base availabilities with a 10% retry budget: at 99% base availability, retries push perceived availability to 99.99%. At 90%, retries get you to 99%. But at 80% base availability, retries only get you to 88%, not because the math breaks, but because a growing share of requests fail even after their one retry, so the budget stops buying anything. And in the failure modes engineers actually deal with, an overloaded service, a bad database host, sharding problems, errors are correlated, not independent. Retrying against a service that’s failing because it’s overloaded doesn’t sample a healthy instance. It adds more load to the same overloaded instance. Past a certain failure rate, the retry is pure overhead, not insurance.
Claim and unclaim
The mechanism Uber built to act on this is a header, x-uber-error-claim, layered on top of a dependency-mapping system they described in a companion post two days earlier. A service that returns an error because one of its own outbound calls failed marks that error “unclaimed” when propagating it upward, meaning: this isn’t mine, don’t retry to me expecting it to help. A service that returns an error without any failed outbound call, meaning the failure genuinely originated there, marks it “claimed.” The caller’s retry logic then does the obvious thing: retry on a claimed error, don’t retry (or retry only once, to guarantee at-least-once semantics) on an unclaimed one.
The correlation is done with ordinary request-scoped context, not a new tracing pipeline. Uber’s yarpc-based inbound middleware assigns a request ID and stores an IngressLog keyed by it in a shared in-memory map; the outbound middleware retrieves that ID from the request context and appends an EgressLog entry recording whether the downstream call failed. By the end of the request, the inbound middleware has a complete record of whether this service’s own error correlates with any of its outbound failures. That correlation, done via metrics rather than distributed tracing, matters for a scale reason: Uber calculated that at 0.01% trace sampling and 99.9% API availability, the odds of sampling a single failed request are 1 in 10 million, meaning a service doing 500 requests per second would take over five hours to capture one failure sample through tracing alone, and over two days to get ten. Metrics observe every request, so the correlation converges fast enough to be useful operationally instead of retrospectively.
Whether an edge is treated as fail-close (a failure here reliably fails the caller) or fail-open (it doesn’t) comes from a simple ratio: the fraction of callee failures where the caller also failed, over all callee failures regardless of caller outcome. Above 0.8, fail-close; below 0.2, fail-open; in between, unknown and left alone. This ratio is what lets the system decide, automatically and per edge, whether an error is worth propagating retry pressure for at all.
The honest edge case
The scheme isn’t free of false positives. If a service is failing for reasons unrelated to any tracked downstream, say a cache or database problem the dependency system doesn’t model, and a genuinely separate downstream also happens to be failing at the same time, the system can misattribute the error to the downstream and unclaim it, suppressing a retry that would have actually helped. Uber quantifies this directly: in the naive case, roughly 1.9 out of every 10 such coincidental failures get incorrectly unclaimed. Using six months of production correlation data as a fail-close memory to break the tie drops that to about 0.4 out of 10. It’s not zero, but it’s a small, measured cost against a system-wide protection.
What it bought them
On November 18, 2025, a core entity service more than five levels deep in Uber’s call graph started failing hard due to an infrastructure issue. Simple retry budgets, applied uniformly at every hop, would have added an estimated 46% to 135% more traffic onto an already-degraded service, the kind of feedback loop that turns a bad incident into a much longer one. Because error ownership was already live, immediate callers were stopped from retrying, with some individual services prevented from making up to 200,000 additional requests each, and an estimated 9.5 million spurious requests suppressed across the whole mesh once the effect was traced back through every ancestor node.
The aggregate metric Uber tracks now is “max retry storm radius”, the deepest point in a call graph where a retry storm could still propagate after the fix. Across all of Uber’s user-facing APIs, the maximum dropped from 25 to 3, and the average from 20 to 2.
The decision this changes
If your service mesh relies on retry budgets alone, the number you should be worried about isn’t the per-hop percentage, it’s the compounded multiple at your deepest fan-out path, and whether your failure modes are actually independent enough for a retry to help rather than pile on. You don’t need Uber’s exact infrastructure to apply the idea: any system with inbound/outbound middleware that can share request-scoped context can build the same claim/unclaim correlation without a new tracing stack, using metrics instead of traces to get fast enough convergence to matter. The part worth stealing outright is the reframe: a retry policy that only asks “how many” is solving the wrong problem. The one that matters is “whose error is this.”
Sources
[1] https://www.uber.com/us/en/blog/protecting-against-retry-storms/: Uber Engineering, “How Uber Protects Against Retry Storms,” September 17, 2026
[2] https://www.uber.com/us/en/blog/automated-dependency-analysis/: Uber Engineering, “Large-Scale Automated Dependency Analysis Across Uber’s Service Mesh,” September 15, 2026
[3] https://github.com/yarpc/yarpc-go/blob/release/v1.86.0/api/middleware/outbound.go#L49: yarpc-go outbound middleware interface referenced in [2]