RPCs, Deadlines & Cancellation
Turn remote calls into bounded work by propagating time budgets and stopping work that no longer matters.
2–5 minute refresher
Client→Service A→Service B→State
One user request carries one shrinking time budget through the entire call graph.
30second
refresher
refresher
An RPC looks like a local function call but crosses independent processes, queues, networks, and failure domains. A deadline states when the result stops being useful; cancellation asks every layer to stop spending resources after that point.
What problem does it solve?
Independent per-hop timeouts can exceed the user's total budget and leave abandoned work running. Under load, that wasted work consumes the exact capacity needed for recovery.
How it works
- Assign the request an absolute deadline and stable operation ID.
- Propagate the remaining budget across RPC metadata.
- Reject work that cannot finish before its deadline.
- Cooperatively cancel queued, running, and downstream work.
- Make retries safe because cancellation and completion may race.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| Short deadline | Bounds latency and resource use | More false timeouts |
| Long deadline | More slow successes | Larger queues and recovery time |
| Hedged request | Reduces tail latency | Extra load and duplicate execution |
What happens if?
The caller leaves but work continues
A timed-out request keeps database queries, model inference, and downstream RPCs running. New requests queue behind results nobody can use, creating a positive feedback loop.
Where it appears
- gRPC and service meshes
- Database drivers
- Workflow engines
- AI inference gateways
Senior interview modeShould every downstream call receive the same timeout?Show answer
No. Each call receives the remaining end-to-end budget, often minus a small reserve. Parallel branches may share the budget; sequential calls must consume it. The deadline is a product contract, not a per-library default.