A wallet product fails traffic from us-east to us-west. Some users see a lower balance and repeat transfers.
Consistency incident · Senior
Users see stale balances after regional failover
Failover restores availability, but asynchronous replicas expose older account state.
Establish the operating contract
Incident brief
One write leader per account region, asynchronous cross-region replication, CDN-cached profile summaries, and clients that may retry against the new region.
A confirmed transfer must not be applied twice, and available funds must never be overstated.
Constraints
- A global write outage longer than five minutes triggers regulatory escalation.
- The ledger is authoritative; cached balance summaries are derived.
- The old region may return with writes not present in west.
Build a causal model
Evidence timeline
East loses power after acknowledging transfers through log position 8,442,190.
West is promoted at position 8,441,760; observed replication lag was reported as 0 seconds just before failure.
Support reports balances moving backward after refresh.
Old east instances intermittently accept health checks as networking returns.
Reason before revealing
Your response
1Diagnose
What does the log-position gap tell you that the lag-in-seconds metric did not?
Need a nudge?+
Freshness is about replicated work and acknowledgements, not wall-clock delay alone.
2Stabilize
How do you prevent two writable primaries while preserving a safe customer experience?
Need a nudge?+
Promotion needs authority, fencing, and an explicit degraded mode.
3Recover
How do you reconcile acknowledged-but-missing transfers?
Need a nudge?+
Separate the immutable ledger from derived balances and client retries.
4Prevent
What failover contract would make RPO visible to the product?
Need a nudge?+
Define readiness using positions, fencing, and business invariants.
Compare reasoning, not wording
Model response
Reveal structured response
The promoted replica is behind despite a misleading time-lag metric. Reads regress, retries can duplicate acknowledged transfers, and the returning region creates split-brain risk unless its old lease or epoch is fenced.
- Fence the old leader using a higher epoch at the write authority and downstream stores.
- Disable or shorten derived caches and mark balances as temporarily reconciling.
- Route transfer retries through idempotent operation IDs; reject writes whose authority epoch is stale.
- If available funds cannot be proven, fail closed for debits while allowing safe reads or deposits according to policy.
- Compare durable ledger positions and recover missing acknowledged records from the old region or transaction journal.
- Deduplicate by transfer ID, replay into the authoritative ledger, then rebuild balance projections.
- Audit every response acknowledged beyond the promoted position and communicate affected accounts.
- Return east only as a follower after state convergence and epoch validation.
- Promotion gates based on log sequence and acknowledged-write watermarks, not lag seconds.
- Consensus-backed or otherwise strongly fenced regional authority.
- Read-your-writes tokens or session watermarks for post-transfer reads.
- Document RPO behavior and rehearse failover with acknowledged-write gaps.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Trusting lag measured only in seconds
- Clearing caches and assuming the data is repaired
- Promoting without fencing the previous leader
- Replaying transactions without stable IDs
Strengthen the underlying concepts