A limited-edition sale has one globally shared unit remaining. Both regions confirm an order during a WAN partition.
Architecture incident · Staff+
Two regions sell the last unit
Active-active inventory remains available through a partition and violates uniqueness.
Establish the operating contract
Incident brief
Active-active APIs write to local databases. Inventory counters replicate asynchronously and conflicts resolve by last-write-wins timestamp.
Confirmed sales must never exceed owned inventory for scarce items.
Constraints
- Most catalog items should remain locally writable.
- Scarce releases may tolerate queuing or a brief rejection.
- Wall clocks are synchronized only within tens of milliseconds.
Build a causal model
Evidence timeline
Both regions read available=1 at local revision 912.
Region A confirms order A-17; region B confirms B-42.
WAN health shows 60% packet loss; local health remains green.
Replication resumes and last-write-wins leaves available=0, hiding the double sale.
Reason before revealing
Your response
1Diagnose
Why does last-write-wins hide rather than resolve this conflict?
Need a nudge?+
A converged value does not prove the invariant held across operations.
2Stabilize
What do you do with new orders and the two confirmations?
Need a nudge?+
Protect the invariant, preserve evidence, and define customer remediation.
3Recover
How do you determine the authoritative winner without relying on clocks?
Need a nudge?+
Authority and a serial order must come from the protocol or preallocated rights.
4Prevent
How would you preserve local writes for ordinary items but coordinate scarce inventory?
Need a nudge?+
Consistency can be selected per invariant and key.
Compare reasoning, not wording
Model response
Reveal structured response
Two local read-modify-write operations consumed the same logical unit. Last-write-wins makes replicas converge but cannot restore the violated conservation invariant; timestamps do not create exclusive authority.
- Stop or queue scarce-item sales in both regions and preserve both order histories.
- Choose an authoritative order using a predeclared sequencer or durable reservation order—not ad hoc wall time.
- Honor or compensate customers under an explicit business policy.
- Audit similar SKUs because the merged counter value conceals oversells.
- Reconstruct operations from immutable order and reservation records.
- Recompute inventory from grants and consumptions rather than trusting the converged counter.
- Issue cancellation/compensation through idempotent workflows.
- Resume only after one authority or safe regional rights are established.
- Route scarce-key reservations through a consensus-backed home region or global sequencer.
- Alternatively escrow inventory rights to regions so local decrements cannot exceed regional grants.
- Fence authority with epochs and reject stale writers.
- Use invariant-oriented reconciliation that detects conservation violations, not only replica divergence.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Increasing clock synchronization accuracy
- Declaring the later timestamp the morally correct order
- Checking only that replicas eventually match
- Making the entire catalog globally serializable without product analysis
Strengthen the underlying concepts