Introduction

Retry storms historically impact business operations and brand trust. While retry configuration tuning and retry budgets provide meaningful mitigation at the service level, they’re manually configured and lack visibility into cross-service amplification caused by deep dependency chains and fan-out patterns. As a result, it can be difficult to shield infrastructure against the domino effect triggered by a single service outage deeper in the stack. 

A key reason is that retry behavior today isn’t context-aware. While we can control how many retries occur, we can’t precisely control when they occur. This stems from the challenge of reliably distinguishing between errors generated by a service and those merely propagated through it.

As a result, retries are applied uniformly rather than conditionally.

This approach works for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience.

One might argue that error codes from downstream services could be translated upstream to provide context for retries. While theoretically possible, this approach doesn't scale at Uber due to large fan-in and fan-out, evolving call flows, and the need for frequent adaptive changes. Therefore, we developed a context-aware mechanism in shared infrastructure to handle errors more efficiently. This blog explains the mechanism.

Background

Consider a simple call chain as shown in Figure 1, where the total number of requests arriving at Node_A is Ƞ. By deduction, all nodes B, C, D, E, F, and G serve Ƞ requests in the steady state (when no node errors out).

Figure 1: Call-chain with 1:1 fan-out, where a node calls its downstream exactly once for any incoming request.

If service D starts erroring out and each service is configured to retry once (1 regular attempt and another attempt if the downstream fails), let’s look at the total number of requests served by each node.

Figure 2: Call chain where a service errors out.

This can be distilled down to a simple formula, assuming the number of retries R is the same at every hop. ɗ denotes the depth of the node in the call chain, the number of requests served by the node if the node creates or passes through an error:

Rɗ × Ƞ

Retry Budgets

We can optimize this by introducing retry budgets. Let’s assume the same retry budget at every hop represented by B. The new formula becomes:

(1+B)ɗ × Ƞ

Now, let’s try to see the number of requests served with a retry budget of 10%:

In the above example, the error originates at Node_D, and if we limit the retry to only between Node_D and Node_C , and restrict entirely Node_A and Node_B  from retrying on this error, we can guarantee a similar availability of the call-path without overburdening Node_D, Node_E,  Node_{F ,} and  Node_G.

Error Ownership

Consider the same example of retry budgets while restricting retries between the edge from Node_C  to Node_D, where the error originates.

Here, we clamp down the total number of requests served by all nodes from D till the leaf node G to just 10% over baseline, while allowing at least once retry for up to 10% of errors when they’re first returned. But what about the availability of Node_D as seen by Node_C? Let’s run some numbers for various availability scenarios, and try to calculate ‌availability after retry.

As shown in the table above, for availability drops up to 10% in the callee node, even a single retry is helpful in getting the perceived availability by the caller node up to 99%. Beyond this, perceived availability drops significantly as a good chunk of requests are never retried due to the retry budget in place.

This calculation assumes the errors from the callee are independent and that retries will lead to recovery. However, in many real-world scenarios like service overload, bad database hosts, database overload, or sharding issues, the probability of retries remains high even with retries. 

This contradicts the idea that retries to callee always increase perceived availability to the caller. It’s also this intuition that forms the basis of error ownership. During periods of high error rates from a service, the errors are less likely to be randomized, and wouldn’t benefit from a higher number of retries, and instead might be responsible for further degradation.

Architecture

The solution is about establishing error ownership, which can be explained using the symptom versus cause analogy.

If a service calls N outbounds for fulfilling a request, and if an outbound error-out causes it to return an error, then the error returned by that service is only a symptom. Simultaneously, in the context of the service, the cause is the incoming error from its downstream.

However, if no outbound of the service errors out while fulfilling the request and it still returns an error, the service is the cause of the returned error, and is the owner. In the next section, we discuss some possible solutions that can leverage this.

Simple Correlation

Claiming Error Ownership

We use the Service Dependency Analysis Solution to correlate an inbound failure with an outbound failure and use the ruleset shown in Figure 3 for making or refuting error claims.

Figure 3: Decision logic for claiming error ownership.

Retrying with Error Ownership

The caller uses the logic shown in Figure 4 to determine if it should retry the request. 

Figure 4: Decision logic for allowing retries.

Here, the first node to see a missing error claim from a downstream unclaims the error, limiting the impact radius of the retry disturbance (it’s no longer a storm), while still allowing sufficient retries to the error-returning service.

Decision Matrix

Figure 5: 3 nodes used to demonstrate caller errors.

Figure 6: Decision logic for Error claim propagation.

Coincidental Errors and Why We Need Service Dependency Analysis

The decision matrix above covers cases where the downstream call fails or there’s an internal server error. There could also be scenarios where both happen simultaneously, as shown in Figure 7.

Figure 7: Example where Node A has 2 fail-open dependencies, Node B and Node C.

The service dependency analysis solution tries to attribute errors to downstreams first and itself last, so even when Nodes B and C are fail-open, it assigns the blame to them when the failures are colocated. This unclaims the error and prevents the upstream of Node A from retrying to it and possibly recovering.

The impact of missed legitimate retry opportunities in such a scenario is significant. If Node A serves 100 requests, out of which 10 experience internal server errors originating at Node A, 1.9 of those 10 requests would be incorrectly unclaimed by Node A as an error not originating from itself, resulting in a missed legitimate retry opportunity.

Based on the analysis of the 6 months of service dependency analysis solution data, coincidental errors like a legitimate server error or a fail-close dependency error happening during a fail-open dependency error are extremely rare. In the worst case scenario, for 80% of edges with over 100 callee failures in a minute, around 2% of the times the caller failed too (a coincidental failure). Using the example above, we’d get 0.396 out of 10 requests that’d be incorrectly unclaimed by Node A.

However, even if we fixate on the worst case, since the service dependency analysis solution creates a memory of failure patterns, it can leverage this memory to only unclaim errors when inbound failures correlate with outbound failures in fail-close dependencies. This eliminates the risk of retry suppression during coincidental errors.

Figure 8: Decision logic for determining when to retry.

Edge Case Scenarios

Guaranteeing At-Least-Once Retries

Many services don’t have retries configured for their fail-close outbounds. Eliminating retries from callers of these services would lead to availability drops along the incoming caller chain. 

This is solved by introducing a flag to signal whether retry criteria is satisfied for an error returned by a downstream. When the retry middleware sees an error from the downstream, it can compute whether the retry criteria is satisfied and pass that along to the service dependency analysis solution. 

Figure 9: Retry logic to guarantee at-least-once retries.

Figure 10: Decision logic to determine whether the retry criteria is satisfied.

Figure 11: Decision logic for service error ownership.

Figure 12: End-to-end flow.

By introducing an at-least-once-retry guarantee, we eliminated availability drops in our call chains. At the same time, we safeguarded them against retry storms by leveraging error ownership.

Context Drop Handling

Figure 13: 4 nodes used to demonstrate context drop handling.

As a result, Node C claims the error, while decoupled retries at the retry middleware continue to happen between Node C and D. A set of retries also happens between Node B and C, as Node C must claim the error it’s returning. However, when the retries fail and Node B returns an error to Node A, it also returns the unclaimed error header, which should prevent Node A from retrying the request to Node B. In this case, we cut down the total number of requests to Nodes B through D by half, assuming a single retry (1 attempt and 1 retry) configuration at all nodes. If there were 5 nodes to the left of Node A, the worst case would’ve had 32 times more requests without error ownership propagation.

Figure 14: Description.

Error Ownership in Production

Everything described so far isn’t theoretical—error ownership is fully implemented and operational across Uber’s service mesh today. The scheme runs in the retry middleware and the Service Dependency Analysis Solution that sit in the request path of our user-facing APIs, continuously claiming and unclaiming errors as traffic flows through deep dependency chains. Because it’s embedded in shared infrastructure, services inherit retry-storm protection without bespoke per-service error-handling logic. The following real-world incident demonstrates how this production deployment behaved under a genuine large-scale degradation.

Use Cases at Uber

On November 18th, 2025, Uber had a major outage due to an issue with a Core Entity service. The service lives over 5 levels deep in our call chain and is critical for business operations. It started returning a very high error rate due to an underlying infrastructure issue. This error was quickly propagated up the call chain. During this time, many upstream caller services had enough opportunity to retry the failed requests returned by these services. With simple retry budgets, this would’ve resulted in a 46%-135% traffic increase on the degraded service, prolonging the outage by diminishing chances of recovery. Because error ownership was already enabled in production, as described above, the system contained the blast radius automatically

Immediate retry attempts were stopped to the immediate callers of the degraded service, where some callers were stopped from making up to 200,000 additional requests. We also calculated the retries that were stopped at the ancestors of these immediate callers, and aggregated them at the root node. We learned that we could stop a staggering 9.5 million spurious requests in our service mesh. Those could’ve easily prolonged the outage by constantly hammering the degraded service.

Conclusion

Our approach has dramatically reduced the total request volume flowing through the call graph during degradation events.

To quantify this, we define the max retry storm radius after error ownership as the max depth of call path where a retry storm could happen after error ownership was enabled. It’s computed for the call graph of every root node. Across all our user-facing APIs, we got this value down to a maximum of 3, where the earlier value of max retry storm radius was up to 25. We also got the average down to 2 from 20.

Figure 15: Max retry storm radius after error ownership.

Acknowledgments

Cover Photo Attribution: Generated with ChatGPT by OpenAI; no external images, logos, or third-party assets used.

Stay up to date with the latest from Uber Engineering—follow us on LinkedIn for our newest blog posts and insights.