Reliability · Incident commander
Checkout is melting under a retry storm
A slow payment dependency turns ordinary retries into a cascading failure across checkout.
Failure and retry patternsCircuit breakersLoad shedding
Enter scenario →Make the call
under pressure.
Work through realistic incidents with incomplete evidence and competing constraints. Each exercise asks you to diagnose the failure, stabilize the system, recover safely, and prevent recurrence.
8deep scenarios4reasoning phases10rubric points each
Scenario library
Choose an incident
Showing 8 scenarios
Consistency · Service owner
Users see stale balances after regional failover
Failover restores availability, but asynchronous replicas expose older account state.
Geo-replicationRPO and RTOConsistency models
Enter scenario →Consensus · On-call engineer
A Raft cluster keeps changing leaders
Uneven latency and pauses create election churn without an obvious node failure.
RaftLeader electionQuorums
Enter scenario →Partitioning · Platform engineer
One celebrity account overwhelms a shard
A technically balanced keyspace collapses when one key becomes globally hot.
Hot partitionsConsistent hashingRebalancing
Enter scenario →Messaging · Application architect
Orders are fulfilled twice after a consumer restart
At-least-once delivery meets a non-idempotent warehouse side effect.
Delivery semanticsIdempotent workflowsDistributed logs
Enter scenario →Architecture · Principal engineer
Two regions sell the last unit
Active-active inventory remains available through a partition and violates uniqueness.
Active-activeConflict resolutionPACELC
Enter scenario →Coordination · Distributed systems engineer
A clock jump creates two lease holders
Time-based ownership outlives its safety assumptions after virtualization pauses and clock correction.
Time and orderingLeader electionIdempotent workflows
Enter scenario →Messaging · Data platform owner
A slow sink stalls the event pipeline
Consumer lag, oversized batches, and unbounded buffering turn degradation into data loss risk.
BackpressureConsumer groupsLoad shedding
Enter scenario →No matching scenarios.Try another level, area, or search phrase.