Functional
- Get, put, conditional update, and delete opaque values by key.
- Choose a consistency level per dataset or operation.
- Continue through node, zone, and selected regional failures.
End-to-end walkthrough
Global Key-Value Store
Serve keys near users while balancing consistency, conflicts, and regional failure.
Frame before solving
Requirements & boundaries
Quality attributes
- Local-region p99 under 20 ms for ordinary operations.
- Conflicts are prevented or resolved according to declared policy.
- Repair converges replicas without blocking foreground traffic.
Explicitly out of scope
- Ad hoc secondary queries.
- General multi-key serializable transactions.
Size the important constraints
Back-of-the-envelope estimates
Use stable hash partitioning and continuous rebalancing.
Regional coordinators and replicated partitions scale independently.
Synchronous global quorum is too slow for every dataset.
Metadata for versions/conflicts is a meaningful fraction.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/kv/{key}?consistency=Read with strong, quorum, session, or eventual policy./v1/kv/{key}Write value with version precondition and idempotency token./v1/kv/{key}Write a versioned tombstone retained through repair horizon.Authoritative records
key, value/tombstone, version_vector or HLC, origin, expires_atCausal metadata distinguishes newer from concurrent updates.token_range, epoch, replicas by region/zoneEpoch fences coordinators using stale topology.Trace the critical path
Architecture & request flow
- 1Route to nearest healthy region→
- 2Locate key replicas→
- 3Apply consistency policy→
- 4Replicate across regions→
- 5Repair divergent state
Global router
Select region using health and locality
Regional coordinators
Execute consistency policy
Partition replicas
Store and version keys
Repair workers
Run anti-entropy and reconcile conflicts
Reason about the hard parts
Critical design deep dives
Consistency classes
Strong keys route writes to a home-region consensus group. Local-latency keys accept regional writes and replicate asynchronously, using conditional versions or domain-specific merge for concurrency. Expose semantics as named policies, not raw R/W knobs.
Conflict handling
Last-write-wins is acceptable only when lost concurrent updates are harmless. Sets, counters, and carts can use mergeable structures; account balance or uniqueness requires a single authority or transaction.
Anti-entropy and failover
Merkle/range hashes find divergence, then stream only mismatched keys. During regional failover, advance a placement epoch and fence the old authority. Re-entry happens gradually after repair, not by immediately restoring traffic.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Tunable consistencyMatches latency to operation riskApplication teams must understand semantics
Active-active regionsKeeps writes local during normal operationConcurrent updates need conflict resolution
Failure-first review
What happens if…?
Regions partition
Continue only the operations permitted by the data invariant and record reconciliation metadata.
Clock order is misleading
Use logical or hybrid versions and never use wall time alone for correctness.
Avoid premature complexity
How the design evolves
Regional KV
Hash partitions with quorum replication
Move here when: Low-latency scalable key access.
Geo reads
Async remote replicas and session tokens
Move here when: Global users need local reads.
Active-active classes
Policy-based writes, conflict metadata, anti-entropy
Move here when: Regional write continuity is worth reconciliation complexity.
Test the reasoning
Interview follow-ups
When is last-write-wins unsafe?
Strong answer signal: When concurrent updates carry independent information or clocks cannot establish causality.
How does a client get read-your-writes after changing regions?
Strong answer signal: Carry a session/version token and wait, route, or fail explicitly.
What happens when the old region returns after failover?
Strong answer signal: Fence old epochs, repair divergence, then restore placement deliberately.
Build from primitives