Functional
- Get, set, delete, increment, and multi-get bounded-size values.
- Expire entries and support explicit invalidation.
- Scale shards without remapping the entire keyspace.
End-to-end walkthrough
Distributed Cache
Serve hot data quickly while controlling staleness, skew, and stampedes.
Frame before solving
Requirements & boundaries
Quality attributes
- Single-key hit p99 below 2 ms in-region.
- Cache loss cannot corrupt authoritative data.
- Protect the source during misses, resize, and recovery.
Explicitly out of scope
- Durable primary storage.
- Arbitrary queries or cross-key transactions.
Size the important constraints
Back-of-the-envelope estimates
Client routing and connection reuse matter.
Use many shards with replicas and memory headroom.
Protocol and metadata overhead are material.
A 5-point drop can double source traffic.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/cache/{key}Return value, version, and remaining TTL./v1/cache/{key}Set bounded value with TTL and optional compare-version./v1/cache/{key}Invalidate a key across replicas.Authoritative records
key, value, version, expires_at, sizeVersion enables safe invalidation and stale-write rejection.epoch, token_range, primary, replicasClients cache this control-plane state and reject stale epochs.Trace the critical path
Architecture & request flow
- 1Hash key to shard→
- 2Read cache entry→
- 3Return hit→
- 4Fetch source on miss→
- 5Populate with bounded TTL
Cache clients
Route, retry, and coalesce requests
Cache shards
Hold ephemeral key/value state
Source of truth
Provide authoritative values
Invalidation stream
Remove or refresh changed entries
Reason about the hard parts
Critical design deep dives
Placement and movement
Use consistent hashing with virtual nodes and weighted capacity. During movement, read old then new placement and write both until an epoch cutover; rate-limit migration to protect foreground traffic.
Stampede control
Coalesce concurrent misses per key, jitter TTLs, refresh hot entries before expiry, and serve slightly stale data when the source is unhealthy.
Consistency
For mutable objects, write the database first, then publish versioned invalidation. A delayed older event must not delete a newer cached value. Strong read-after-write should bypass or update cache explicitly.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Consistent hashingLimits movement during resizingVirtual-node balance needs monitoring
TTL plus eventsBounds staleness and accelerates correctionTwo invalidation mechanisms to operate
Failure-first review
What happens if…?
Hot key expires
Use request coalescing, jittered TTLs, and stale-while-revalidate.
Shard disappears
Remap keys gradually and protect the source from the miss surge.
Avoid premature complexity
How the design evolves
Application cache
Local LRU with TTL
Move here when: One process and small working set.
Shared cache
Partitioned regional fleet with replicas
Move here when: Many service instances need a common working set.
Critical platform
Epoch-aware rebalancing, request coalescing, stale serving
Move here when: Cache failures threaten primary stores.
Test the reasoning
Interview follow-ups
What happens when a 20% cache fleet loss remaps keys?
Strong answer signal: Discuss controlled movement, spare capacity, and source protection.
How do you invalidate safely when events reorder?
Strong answer signal: Version every mutation and ignore stale invalidations.
Why can a high global hit rate still hide a problem?
Strong answer signal: Segment by tenant, key class, shard, and source cost.
Build from primitives