Checkout handles 18k requests per second during a flash sale. Abandoned carts are rising by 4,000 per minute.
Reliability incident · Senior
Checkout is melting under a retry storm
A slow payment dependency turns ordinary retries into a cascading failure across checkout.
Establish the operating contract
Incident brief
API gateway → checkout service → inventory reservation and payment authorization. Each layer retries twice with exponential backoff, but deadlines are not propagated.
A customer must never be charged twice, and confirmed inventory must not be silently lost.
Constraints
- The payment provider cannot be changed during the incident.
- The business will accept degraded conversion before duplicate charges.
- Inventory reservations expire after 12 minutes.
Build a causal model
Evidence timeline
Payment p99 rises from 380 ms to 4.8 s; error rate remains below 2%.
Checkout concurrency triples and outbound payment traffic reaches 2.7× incoming traffic.
Thread pools saturate; healthy inventory calls now time out behind payment work.
Payment recovers, but checkout remains overloaded and queues continue growing.
Reason before revealing
Your response
1Diagnose
What evidence distinguishes dependency slowness from insufficient checkout capacity?
Need a nudge?+
Compare arrival rate, useful completions, retry amplification, queue time, and saturation.
2Stabilize
Which controls do you apply first, and in what order?
Need a nudge?+
Protect the invariant and stop amplification before adding capacity.
3Recover
How do you resolve timed-out payments and inventory reservations safely?
Need a nudge?+
A timeout is an unknown outcome, not a failed operation.
4Prevent
What service contract and telemetry would keep this failure bounded next time?
Need a nudge?+
Think deadlines, budgets, admission control, operation identity, and recovery probes.
Compare reasoning, not wording
Model response
Reveal structured response
This is retry amplification plus queue collapse. A slow dependency increases in-flight work; layered retries multiply it; expired callers leave useless work consuming capacity. The fact that checkout stays unhealthy after payment recovers points to backlog and synchronized retries, not a permanent capacity shortage.
- Freeze nonessential changes and declare duplicate charge prevention the protected invariant.
- Disable or sharply budget retries at one controlled layer; propagate remaining request deadlines.
- Cap payment concurrency and shed low-priority or already-expired checkout work.
- Open a circuit for new payment attempts while allowing sparse, jittered recovery probes.
- Reconcile every ambiguous authorization by stable payment-attempt ID before retrying.
- Return recorded outcomes for duplicate idempotency keys instead of reapplying side effects.
- Drain or discard expired queue entries, then ramp admission gradually while watching useful throughput.
- Reconcile inventory holds against durable payment outcomes before releasing or confirming them.
- One end-to-end retry budget, not independent retries at every hop.
- Persist idempotency key and payment outcome atomically with checkout state.
- Alert on retry amplification, queue age, deadline-exceeded work, and useful completion rate.
- Test slow dependencies and recovery waves with load, not only hard failures.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Blindly increasing retry count
- Treating every timeout as a failed payment
- Autoscaling checkout without limiting downstream concurrency
- Closing the incident as soon as payment latency recovers
Strengthen the underlying concepts