Start with the invariant, then choose the weakest model that preserves it.
Partition occurs ↓ Preserve one truth or keep every side responsive ↓ During normal operation: consistency or lower latency
Rules of thumb CAP describes partition-time behavior. PACELC adds the normal-time latency trade-off. R + W > N creates overlap, not automatic linearizability. Session guarantees often solve user-facing needs without global coordination. Failure warning Do not call a system strongly consistent without naming the operation, scope, and failure assumptions.
Open the full lesson → Consensus establishes one authoritative sequence despite failures and delay.
Elect ↓ Propose ↓ Replicate ↓ Majority ↓ Commit ↓ Apply
Rules of thumb A majority of N is floor(N/2) + 1. Terms or epochs distinguish current authority from stale leaders. A committed entry survives the failures covered by the quorum. Fencing tokens make former leaders harmless. Failure warning Failure detection can justify replacement, but cannot prove the old leader stopped.
Open the full lesson → The partition key determines placement, parallelism, locality, and hot spots.
Choose key ↓ Map ownership ↓ Route ↓ Measure skew ↓ Split or move ↓ Fence old owner
Rules of thumb Hashing spreads point traffic but loses ordered locality. Ranges preserve scans but can create leading-edge hot spots. Consistent hashing limits movement when membership changes. Average cluster capacity cannot fix one indivisible hot key. Failure warning Rebalancing is a live migration: copy, catch up, switch authority, then remove the old copy.
Open the full lesson → Choose atomic coordination or an explicit workflow based on the business invariant.
Define invariant ↓ Choose commit boundary ↓ Handle ambiguity ↓ Retry by identity ↓ Compensate or reconcile
Rules of thumb 2PC can block prepared participants while the decision is unknown. A saga commits locally and compensates business effects. MVCC snapshots remove many read/write conflicts, not every anomaly. Idempotency requires one stable operation identity and durable outcome. Failure warning Compensation creates a new event; it does not erase what users or external systems already observed.
Open the full lesson → Optimize the amplification dimension your workload can least afford.
Write ↓ WAL ↓ Memtable ↓ Flush SSTable ↓ Read across runs ↓ Compact
Rules of thumb LSM trees turn random writes into sequential I/O. Bloom filters avoid many absent-key disk reads. Compaction trades background writes for bounded reads and space. Long snapshots delay version garbage collection. Open the full lesson → Separate broker delivery from the end-to-end business outcome.
Append ↓ Partition ↓ Consume ↓ Apply idempotently ↓ Commit offset ↓ Replay safely
Rules of thumb Ordering is normally scoped to one partition. At-least-once delivery makes duplicates normal. Parallelism in one consumer group is bounded by partition count. A queue absorbs a burst; it cannot fix sustained overload. Failure warning Exactly-once claims end at a specific atomic boundary; external side effects need their own protection.
Open the full lesson → Bound work before a struggling dependency turns into a cascading failure.
Deadline ↓ Backoff ↓ Jitter ↓ Retry budget ↓ Circuit breaker ↓ Load shedding
Rules of thumb A timeout is an ambiguous outcome, not proof of failure. Retries consume the same constrained capacity as original traffic. Circuit breakers protect resources; they do not repair dependencies. Reject before expensive work when capacity is no longer safe. Failure warning Unbounded queues convert overload into growing latency, memory pressure, and expensive recovery.
Open the full lesson → State write authority, partition behavior, and recovery objectives per operation.
Place replicas ↓ Route traffic ↓ Assign authority ↓ Replicate ↓ Fail over ↓ Reconcile and fail back
Rules of thumb Synchronous remote durability spends wide-area write latency. Asynchronous replication accepts a nonzero RPO. RTO measures restoration time; RPO measures acceptable data loss. Last-write-wins is a policy for discarding one concurrent intent. Open the full lesson →