A social feed normally serves 1.2M reads per second. One livestream drives 180k requests per second to a single creator.
Partitioning incident · Intermediate
One celebrity account overwhelms a shard
A technically balanced keyspace collapses when one key becomes globally hot.
Establish the operating contract
Incident brief
Posts and counters are partitioned by creator ID using consistent hashing. Each partition has one write leader and two read replicas.
Published posts must remain ordered for the creator; view counts may be approximate for up to five minutes.
Constraints
- Per-creator post order is externally visible.
- Clients cannot be updated during the event.
- Approximate counters are acceptable; missing posts are not.
Build a causal model
Evidence timeline
Fleet CPU remains 42%, but shard 317 reaches 100% CPU and 40× normal queue depth.
Adding nodes moves ordinary ranges but creator 8842 remains on one partition.
Replica reads help posts, while counter writes still saturate the leader.
Cache hit rate drops as clients request many slightly different feed windows.
Reason before revealing
Your response
1Diagnose
Why did rebalancing fail despite spare fleet capacity?
Need a nudge?+
A hash function spreads many keys; it cannot divide one indivisible key.
2Stabilize
How do you split reads and writes according to their semantics?
Need a nudge?+
The post log and engagement counters do not require the same consistency.
3Recover
How do you migrate the hot key without breaking order?
Need a nudge?+
Use explicit routing state, epochs, and a cutover watermark.
4Prevent
What hot-key mechanism should exist before the next event?
Need a nudge?+
Detection and a reversible exception path matter more than perfect prediction.
Compare reasoning, not wording
Model response
Reveal structured response
The distribution is balanced by key count but not by load. One creator key is indivisible under the current scheme, so ordinary ring rebalancing cannot spread its work.
- Cache canonical post pages and coalesce identical misses.
- Fan read traffic across replicas while preserving a clear freshness contract.
- Buffer and aggregate commutative counter updates, shedding only nonessential precision.
- Apply per-tenant admission limits so one creator cannot consume recovery capacity.
- Introduce a routing exception for the hot creator rather than reshuffling the entire ring.
- Keep ordered posts on one sequencer or partition substreams with an explicit merge sequence.
- Cut over using an ownership epoch and watermark; reject writers using stale routing.
- Drain aggregated counter deltas idempotently after the event.
- Detect load by key, not only by node or partition.
- Support hot-key replication, request coalescing, and exception routing as tested control-plane operations.
- Separate workloads by invariant: ordered posts, cacheable reads, and commutative counters.
- Capacity-plan for concentration and enforce tenant fairness.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Adding nodes and expecting the hot key to split
- Randomly salting all creator data
- Serving unlimited stale cache without a contract
- Migrating ownership without fencing old writers
Strengthen the underlying concepts