Rebalancing
Move ownership safely as data, traffic, and cluster membership change.
2–5 minute refresher
Leaderaccept writes
replicate ↓
Follower A
Follower B
Follower C
Copy → catch up → switch authority → remove old copy
30second
refresher
refresher
Rebalancing transfers partitions when nodes join, leave, fill up, or become unevenly loaded. A safe move preserves reads and writes while ownership changes under live traffic.
What problem does it solve?
Static placement decays as datasets grow and workloads shift. Simply changing the owner can lose recent writes, produce two authorities, or overwhelm the network.
How it works
- Choose a target using capacity, fault-domain, and load constraints.
- Snapshot and copy the partition to the target.
- Stream changes that occurred during the copy.
- Fence the old owner and atomically publish new routing metadata.
- Verify the target, retain a rollback window, then delete the old replica.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| Fast movement | Correct skew quickly | Network and disk interference |
| Slow movement | Protect foreground latency | Long exposure to imbalance |
| Automatic balancing | Continuous correction | Risk of oscillation without damping |
What happens if?
The source fails halfway through a move
The control plane must know which copy is authoritative and whether the target has a complete checkpoint. Resume from durable progress or restart from another replica; never infer authority from who has more bytes.
Where it appears
- Cassandra streaming
- Kafka partition reassignment
- CockroachDB replica queues
- Elasticsearch shard relocation
Senior interview modeHow do you prevent a rebalancer from causing an outage?Show answer
Use bounded concurrency and bandwidth, health-aware admission, stable placement constraints, progress checkpoints, rollback, and automatic pause conditions tied to foreground latency and saturation.