Kafka & Distributed Logs
Use partitioned append-only logs for durable event streams, replay, and independent consumer progress.
2–5 minute refresher
P0104105106107
P181828384
Consumer group A · offset 106Consumer group B · offset 82
Append to a partition → replicate → advance offset → consumers replay independently
30second
refresher
refresher
A distributed log stores immutable records in ordered partitions. Producers append records, brokers replicate them, and consumer groups track offsets. Retention decouples consumption from production, enabling replay and multiple independent views of the same events.
How it works
A record key selects a partition. One consumer in a group owns each partition at a time, which preserves partition order while spreading work. Acknowledgement policy determines how many replicas must persist an append before success.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| More partitions | More parallel producers and consumers | More metadata, files, rebalances, and no cross-partition order |
| Long retention | Replay, audit, and new consumers | Storage and recovery duration |
| At-least-once | Avoid silent loss after crashes | Consumers must be idempotent |
What happens if?
A consumer processes an event then crashes before committing its offset
The event is delivered again after restart. The handler needs an idempotency key, deduplication record, or transactional output-and-offset commit.
What happens if?
One key produces 80% of traffic
All records for that key land on one partition to preserve order. Adding consumers does not help; the design must split the key, relax ordering, or isolate the hot workload.
Senior interview modeDoes Kafka guarantee global ordering within a topic?Show answer
No. Kafka preserves order within each partition. Global order would require routing through one ordering boundary, which limits parallelism.