A configuration service backs service discovery for 600 applications. Writes stall every few minutes.
Consensus incident · Intermediate
A Raft cluster keeps changing leaders
Uneven latency and pauses create election churn without an obvious node failure.
Establish the operating contract
Incident brief
Five Raft voters across three availability zones; two nodes share a noisy host class. Election timeout is 500–800 ms and heartbeats are 100 ms.
Only committed configuration may be served, and two leaders must never commit conflicting entries in one term.
Constraints
- No voter may be removed until quorum risk is evaluated.
- Configuration reads can tolerate brief staleness but not fabricated state.
- A rolling restart is possible only after the cluster stabilizes.
Build a causal model
Evidence timeline
Leader changes from n2 to n4; no process restart is recorded.
n2 reports a 1.1 s runtime pause and n3 reports 900 ms disk fsync.
Terms increase twice; commit index advances slowly but never diverges.
Network median is 8 ms, while cross-zone p99 briefly exceeds 700 ms.
Reason before revealing
Your response
1Diagnose
Why can healthy processes still cause elections?
Need a nudge?+
Raft reacts to missing timely communication, not the root cause of delay.
2Stabilize
What changes reduce churn without masking a genuinely dead leader?
Need a nudge?+
Compare observed tail latency and pauses with the election window.
3Recover
How do you verify safety while leadership changes?
Need a nudge?+
Terms, commit index, quorum membership, and log matching are your evidence.
4Prevent
Which SLOs and placement rules belong around the consensus group?
Need a nudge?+
Consensus depends on disks, runtime pauses, networks, and failure independence.
Compare reasoning, not wording
Model response
Reveal structured response
Heartbeat processing is delayed beyond the election timeout by runtime pauses, fsync stalls, and cross-zone tail latency. Followers begin elections even though processes are alive. Raft preserves safety, but churn destroys write availability.
- Stop membership changes and confirm a majority shares one committed prefix.
- Move or isolate the noisy workloads if reversible; reduce load on the control plane.
- Increase randomized election timeouts above measured worst-case healthy pauses while keeping heartbeat separation.
- Prefer the most up-to-date, stable candidate and monitor term rate and commit progress.
- Verify term, leader, commit index, last-applied index, and log match across every voter.
- Catch lagging followers up before restarting or changing membership.
- Roll remediation one voter at a time, preserving a known healthy majority.
- Confirm client writes use leader redirection and bounded retry behavior.
- Alert on elections per hour, fsync p99, runtime pauses, heartbeat delay, and commit latency.
- Use failure-independent placement and dedicated resource budgets for voters.
- Benchmark timeout values under realistic pauses rather than median network latency.
- Use pre-vote/check-quorum features where supported and tested.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Restarting all voters together
- Assuming low median latency means the network is healthy
- Removing a slow voter before calculating quorum
- Treating leader churn as evidence of conflicting committed logs
Strengthen the underlying concepts