Functional
- Read and write keys or rows through a stable logical schema.
- Support atomic multi-row transactions and snapshot reads.
- Repartition and repair online.
End-to-end walkthrough
Distributed Database
Provide a coherent data model across partitions, replicas, transactions, and failures.
Frame before solving
Requirements & boundaries
Quality attributes
- Committed data survives tolerated node/zone failure.
- Serializable or clearly documented isolation.
- Predictable behavior during leader movement and overload.
Explicitly out of scope
- Every SQL feature and workload shape.
- Cross-cloud active-active writes in the first version.
Size the important constraints
Back-of-the-envelope estimates
Shard metadata and balancing are control-plane systems.
Gateways cache routing; shards scale independently.
Co-locate common access paths and optimize the fast path.
Write latency includes a quorum round trip.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/transactionsBegin transaction and return transaction identity/snapshot./v1/transactions/{id}/commitConditionally commit writes; retry-safe by transaction ID./v1/rows/{key}?at=Read latest or a specified consistent snapshot.Authoritative records
key+timestamp, value/tombstone, transaction_idVersions support snapshots and conflict checks.start_key, end_key, epoch, replica_set, leaderEpoch fences stale routing and ownership.Trace the critical path
Architecture & request flow
- 1Route key or range→
- 2Coordinate transaction→
- 3Replicate mutations→
- 4Commit authoritative state→
- 5Serve consistent reads
SQL or KV gateway
Parse requests and route work
Transaction coordinator
Track timestamps and commit state
Shard replicas
Store and agree on ordered mutations
Metadata plane
Own placement and rebalancing
Reason about the hard parts
Critical design deep dives
Single-shard commit
A gateway routes to the shard leader. The leader validates conflicts, appends the mutation to its consensus log, waits for quorum, applies to MVCC state, then acknowledges. Followers serve only reads allowed by the consistency contract.
Multi-shard transactions
Use a transaction record plus prepare/commit across participant shards. Resolve an uncertain coordinator by consulting durable transaction status. Timestamps and locks/intents require cleanup and deadlock policy.
Online repartitioning
Snapshot a key range to new replicas, stream the log tail, then transfer authority with a higher range epoch. Gateways encountering stale descriptors refresh and retry safely.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Consensus per shardProvides one authoritative orderWrites pay quorum latency
MVCCAllows nonblocking snapshotsOld versions require cleanup
Failure-first review
What happens if…?
Coordinator response is lost
Expose an idempotent transaction identity and resolve outcome from durable state.
Shard is rebalancing
Transfer authority explicitly and fence stale replicas.
Avoid premature complexity
How the design evolves
Replicated shard
One consensus group + MVCC
Move here when: Prove durability and transaction semantics.
Distributed ranges
Routing metadata, splits, balancing, transaction records
Move here when: Data or write throughput exceeds one group.
Operational maturity
Admission control, backups, follower reads, geo placement
Move here when: Predictability and disaster recovery become requirements.
Test the reasoning
Interview follow-ups
What happens after commit succeeds but the response is lost?
Strong answer signal: Retry with transaction ID and return the durable prior outcome.
How do follower reads stay correct?
Strong answer signal: Read index/lease, bounded staleness, or explicit snapshot timestamps.
How does a split avoid writing to both old and new owners?
Strong answer signal: Epoch fencing and an atomic authority transition.
Build from primitives