Failure & Retry Patterns
Recover from transient failure without turning retries into a synchronized overload event.
2–5 minute refresher
Timeout→Backoff→Jitter→Retry budget→Circuit breaker→Load shedding
Bound waiting → retry less often → randomize → cap work → shed load
30second
refresher
refresher
Reliable remote calls need bounded timeouts, exponential backoff, jitter, idempotency, retry budgets, and overload protection. Retries help transient faults only when the downstream system has capacity to recover.
How the control loop works
Timeouts bound resource occupancy. Backoff spaces repeated attempts. Jitter prevents clients from synchronizing. Retry budgets cap amplification. Circuit breakers stop known-bad calls, while bulkheads and load shedding preserve critical capacity.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| Aggressive retry | Fast recovery from rare packet loss | Load amplification and duplicate work |
| Circuit breaker | Stops repeated calls to a failing dependency | Can delay recovery if thresholds are poor |
| Load shedding | Preserves useful work under saturation | Explicitly rejects lower-priority requests |
What happens if?
Ten thousand clients retry after the same timeout
Without jitter they return together, creating a retry storm that prevents the dependency from recovering. Randomized backoff spreads demand across time.
What happens if?
Three service layers each retry three times
One user request can create up to 27 downstream attempts. Retry at one well-chosen layer and propagate a shared deadline so deeper services cannot outlive the request budget.
Senior interview modeWhy should retries have a budget?Show answer
Retries consume the same constrained capacity as original traffic. A budget limits amplification, protects recovery, and forces the system to fail deliberately rather than extend an outage.