Partial Failures
Design for components and network paths failing independently while the rest of the system continues.
2–5 minute refresher
Client→Service A→Service B→State
Healthy caller + uncertain network + independently failing dependency
30second
refresher
refresher
A partial failure leaves some nodes or communication paths working while others are slow, unreachable, or returning errors. The surviving system must decide how to respond without complete knowledge.
What problem does it solve?
Unlike a local process crash, remote failure is ambiguous. The request, the response, or only one network direction may be lost, and different observers can reach different conclusions.
How it works
- Define a deadline for every remote operation.
- Classify failures as definite, ambiguous, or policy rejection.
- Carry stable identity across retries.
- Isolate the dependency with concurrency and resource limits.
- Reconcile uncertain outcomes from authoritative durable state.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| Short deadline | Protect caller resources | More false timeouts |
| Fail open | Continue service | Possible invariant violation |
| Fail closed | Preserve safety | Reduced availability |
What happens if?
Only responses are dropped
The server successfully applies requests while callers time out and retry. Without idempotency this duplicates effects and makes the healthy server appear broken.
Where it appears
- Remote APIs
- Database clients
- Service meshes
- Multi-zone clusters
Senior interview modeHow do you distinguish a slow dependency from a failed one?Show answer
You cannot know perfectly within a finite deadline. Treat the timeout as a policy decision, use health evidence and repeated observations, and design the operation to remain safe if the dependency was merely slow.