I had a former colleague who would put a “retry 10x with sleeps” in each of A, B, C, D.
None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.
I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
I’d like to mention it since Uber has a really cool load shedder [1], also implemented similarly by Netflix [2] and failsafe-go [3]. It basically looks for points where more requests per second suddenly cause a significant increase in latency and calls that the concurrency limit.
429 (and sometimes 503) errors returned by servers might well be a symptom of intentional load shedding. Perhaps it's just not explicitly called out as a server behavior that induces client retries.
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
I get a feeling of dejavu for this one. Most of my experience has been in Java and in most places I worked in the past we had this hierarchy of exception classification which gets reflected into the http status codes as well. On high level the HTTP status codes in case of errors are already classified as re-tryable or not, the convention varies globally but can be adopted in a standard manner within a company.
The reason why I brought up the exception propagation is cause within a large enough service with multiple layers of depth the exception hierarchy provides with similar context.
The hard part is not implementing something like this, its about maintaining it consistently across every new change. With small product teams this architecture concept/convention/constraint can easily get lost/forgotten and what you are left with is a theoretical system which does not works as desired when the storm comes
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
This can't really be tolerated in practice, though, because it means that one bad component somewhere in your stack, one that is able to accept and respond to requests but for whatever reason isn't able to make requests to its backends, poisons the whole stack. You can't take one backend's word for it that the failure is not localized and therefore retryable.
So many variables, but the simple thing is to set things up like normal rate limiting (which you would want to do anyways). The one generating the errors passes back a retry time. You can add jitter here, tell low priority requests to wait longer, etc.
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
At $previousJob we implemented circuit breakers: centralize all requests to the foreign service (every call to service theta went through the service theta client which had some shared state so everything so we could keep track of requests) and then monitor, when error % got above a certain limit start to dump requests to a text file for sending in the future instead of now. And the centralized caller will send one message every time gap (we started at 30s) and as long as that errors out we keep writing.
We did that because otherwise we would get 2x30 second timeouts to a dead service on every user interaction and it made for a terrible user experience. Keeping track and handling it smartly made the average user experience a lot better.
Just limiting your retry budget to 1% of normal rates using a client-local token bucket with no distributed coordination will eliminate the possibility of long-lived retry storms.
One good option that is not (yet?) mentioned here is a deadline for retries. You can cap the request duration by, say, 500ms and pass the remaining time budget to downstream services.
This can be done via an HTTP header and enforced by the middleware.
Not quite. A token. bucket alone would not prevent a retry storm if you have a chain of services A -> B -> C -> D with a failure in D, you'd end up still having A, B & C all performing retries as they cascade through the chain, albeit (yes) at your configured rate-limits (using token bucket), but you'd still end up with an amplification effect.
So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.
This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)
I had a former colleague who would put a “retry 10x with sleeps” in each of A, B, C, D.
None of these were even expected to fail. But the code was buggy as well, so after D being retried 10000 times, it’d eventually give up. Managed to convert a sub-second operation into a half hour affair.
LOL
Meanwhile Google keeps giving me “please wait, do not reload page” walls, so I ctrl-r as rapidly as possible. Or is that the human test and response?
It is anti-bot proof-of-work defence.
Easy. Take a larger cut from the driver for each retry.
I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
I’d like to mention it since Uber has a really cool load shedder [1], also implemented similarly by Netflix [2] and failsafe-go [3]. It basically looks for points where more requests per second suddenly cause a significant increase in latency and calls that the concurrency limit.
1: https://www.uber.com/us/en/blog/cinnamon-using-century-old-t...
2: https://github.com/Netflix/concurrency-limits
3: https://failsafe-go.dev/adaptive-limiter/
429 (and sometimes 503) errors returned by servers might well be a symptom of intentional load shedding. Perhaps it's just not explicitly called out as a server behavior that induces client retries.
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
I get a feeling of dejavu for this one. Most of my experience has been in Java and in most places I worked in the past we had this hierarchy of exception classification which gets reflected into the http status codes as well. On high level the HTTP status codes in case of errors are already classified as re-tryable or not, the convention varies globally but can be adopted in a standard manner within a company.
The reason why I brought up the exception propagation is cause within a large enough service with multiple layers of depth the exception hierarchy provides with similar context.
The hard part is not implementing something like this, its about maintaining it consistently across every new change. With small product teams this architecture concept/convention/constraint can easily get lost/forgotten and what you are left with is a theoretical system which does not works as desired when the storm comes
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?
2 replies →
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
This can't really be tolerated in practice, though, because it means that one bad component somewhere in your stack, one that is able to accept and respond to requests but for whatever reason isn't able to make requests to its backends, poisons the whole stack. You can't take one backend's word for it that the failure is not localized and therefore retryable.
https://aws.amazon.com/blogs/developer/introducing-retry-thr... (2016 -- 10 years ago!)
https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/... (originally published 2020, republished 2026)
https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
https://aws.amazon.com/blogs/developer/announcing-updated-re... (2026)
So many variables, but the simple thing is to set things up like normal rate limiting (which you would want to do anyways). The one generating the errors passes back a retry time. You can add jitter here, tell low priority requests to wait longer, etc.
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
At $previousJob we implemented circuit breakers: centralize all requests to the foreign service (every call to service theta went through the service theta client which had some shared state so everything so we could keep track of requests) and then monitor, when error % got above a certain limit start to dump requests to a text file for sending in the future instead of now. And the centralized caller will send one message every time gap (we started at 30s) and as long as that errors out we keep writing.
We did that because otherwise we would get 2x30 second timeouts to a dead service on every user interaction and it made for a terrible user experience. Keeping track and handling it smartly made the average user experience a lot better.
Depending on the context, circuit breakers
https://en.wikipedia.org/wiki/Exponential_backoff
Yes, exponential back off and jitter are the first things to work on, and good if you don’t have a better signal (like loss of network).
Also, a simple signal status server or queue system helps to keep global state such that everyone doesn’t retry all at once.
If you have a central error rate server you can skip your retry based on the error rate (100% error rate, don’t retry, etc).
https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...
5 replies →
Just limiting your retry budget to 1% of normal rates using a client-local token bucket with no distributed coordination will eliminate the possibility of long-lived retry storms.
One good option that is not (yet?) mentioned here is a deadline for retries. You can cap the request duration by, say, 500ms and pass the remaining time budget to downstream services.
This can be done via an HTTP header and enforced by the middleware.
[flagged]
[flagged]
This feels like trying to reinvent Fibre Channel's flow control mechanism.
Token bucket is all you need
Not quite. A token. bucket alone would not prevent a retry storm if you have a chain of services A -> B -> C -> D with a failure in D, you'd end up still having A, B & C all performing retries as they cascade through the chain, albeit (yes) at your configured rate-limits (using token bucket), but you'd still end up with an amplification effect.
[dead]