Two workers concurrently execute a daily settlement job, producing conflicting ledger exports.
Coordination incident · Staff+
A clock jump creates two lease holders
Time-based ownership outlives its safety assumptions after virtualization pauses and clock correction.
Establish the operating contract
Incident brief
Workers acquire a database lease with an expiry timestamp based on local wall clocks. The resource accepts writes from any worker presenting the lease ID.
At most one settlement worker may mutate a settlement generation.
Constraints
- The job may pause, but overlapping mutations are unacceptable.
- Workers and the database are in different failure domains.
- Some settlement steps call external systems.
Build a causal model
Evidence timeline
Worker A renews its lease through 02:00:28, then pauses for 38 seconds during host migration.
Worker B observes the lease expired and acquires a new lease.
Worker A resumes with its old lease and continues writing.
NTP steps A's wall clock backward by 420 ms; both workers report valid ownership.
Reason before revealing
Your response
1Diagnose
Why can neither worker's local clock prove exclusive ownership?
Need a nudge?+
A paused process cannot observe that its authority expired elsewhere.
2Stabilize
How do you stop stale workers from mutating state?
Need a nudge?+
The protected resource must participate in rejecting old authority.
3Recover
How do you repair work when external steps may already have run twice?
Need a nudge?+
Fence internal writes and reconcile external operations by stable generation IDs.
4Prevent
What lease protocol and test cases make the timing assumptions explicit?
Need a nudge?+
Use monotonically increasing authority, bounded time assumptions, and pause testing.
Compare reasoning, not wording
Model response
Reveal structured response
Lease expiry is observed independently, while the old holder can pause past expiry and resume. Wall-clock correction worsens reasoning, but the fundamental issue is that the resource accepts stale authority.
- Pause settlement dispatch and fence both workers at the mutation store.
- Issue a new monotonically increasing fencing token to one elected worker.
- Require every protected write to carry the token and reject values below the highest seen.
- Record the affected settlement generation for reconciliation.
- Compare writes by fencing token/generation and rebuild a canonical settlement output.
- Reconcile external calls using stable settlement and operation IDs; compensate duplicates by provider state.
- Do not simply delete one worker's records when downstream effects may exist.
- Resume from a checkpoint under a new fenced generation.
- Use a consensus-backed lease/lock service that returns monotonically increasing fencing tokens.
- Use monotonic time for durations while treating wall-clock time as presentation data.
- Make the storage and external-operation gateways validate generation authority where possible.
- Inject long pauses, delayed renewals, partitions, and clock steps in tests.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Only increasing the lease duration
- Trusting synchronized wall clocks as a lock
- Checking the lease once at job start
- Assuming killing the old process reverses its external effects
Strengthen the underlying concepts