Comment by mandevil

At $previousJob we implemented circuit breakers: centralize all requests to the foreign service (every call to service theta went through the service theta client which had some shared state so everything so we could keep track of requests) and then monitor, when error % got above a certain limit start to dump requests to a text file for sending in the future instead of now. And the centralized caller will send one message every time gap (we started at 30s) and as long as that errors out we keep writing.

We did that because otherwise we would get 2x30 second timeouts to a dead service on every user interaction and it made for a terrible user experience. Keeping track and handling it smartly made the average user experience a lot better.