Functional
- Publish ordered records to partitioned topics.
- Retain records for replay and independent consumers.
- Coordinate consumer groups and expose lag.
End-to-end walkthrough
Kafka-like Event Platform
Retain ordered event streams for independent consumers and replayable processing.
Frame before solving
Requirements & boundaries
Quality attributes
- Acknowledged records survive tolerated broker failure.
- High sequential throughput with predictable tail latency.
- Ordering is explicit per partition key.
Explicitly out of scope
- Arbitrary message priorities.
- Exactly-once effects in external databases.
Size the important constraints
Back-of-the-envelope estimates
Batch, compress, and distribute across thousands of partitions.
~3 PB raw before replication; tier older segments.
Per-record overhead requires batching.
Offsets and rebalances are a substantial control-plane load.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/topics/{topic}/recordsBatch publish with key and producer sequence./v1/topics/{topic}/partitions/{p}?offset=Fetch a bounded sequential batch./v1/groups/{group}/offsetsCommit processed offsets by generation.Authoritative records
topic+partition+offset, key, timestamp, headers, payloadOffset is assigned by the leader and never changes.group, generation, members, assignments, committed_offsetsGeneration fences stale members after rebalance.Trace the critical path
Architecture & request flow
- 1Choose partition key→
- 2Append to leader log→
- 3Replicate record→
- 4Advance producer acknowledgement→
- 5Consumers fetch by offset
Brokers
Store and serve partition logs
Controllers
Manage metadata and leader placement
Producers
Batch, partition, and retry writes
Consumer groups
Divide partitions among workers
Reason about the hard parts
Critical design deep dives
Producer correctness
The partition leader assigns offsets and replicates batches to in-sync replicas. Producer ID plus monotonically increasing sequence lets the leader reject duplicate retries and out-of-order batches.
Partitioning
Keys preserve entity order but create hotspots when distributions skew. Partition count sets the parallelism ceiling and affects metadata, recovery, and future re-keying.
Consumer coordination
A coordinator assigns partitions using a generation. Cooperative rebalancing moves only needed partitions. Consumers commit offsets after effects or atomically with platform-managed output when supported.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Partitioned append-only logCombines throughput with local orderPartition count constrains parallelism
Pull consumptionLets consumers control paceClients manage offsets and lag
Failure-first review
What happens if…?
Partition leader fails
Elect an in-sync replica and fence the previous leader.
Consumer falls behind
Scale within partition limits, reduce work, or accept a recovery window.
Avoid premature complexity
How the design evolves
Replicated log
Static partitions and manual offsets
Move here when: Initial event backbone.
Consumer platform
Groups, idempotent producers, schema governance
Move here when: Many teams and replay workflows.
Large scale
Tiered storage, quotas, rack awareness, cooperative balancing
Move here when: Retention and tenant isolation dominate.
Test the reasoning
Interview follow-ups
Can you increase partitions without changing semantics?
Strong answer signal: Existing key order splits at the change; consumers and partitioners must account for it.
What defines an in-sync replica?
Strong answer signal: Lag/time bounds plus leader epoch; avoid acknowledging replicas that cannot take over safely.
How do you process and publish exactly once?
Strong answer signal: Atomic offset+output transaction inside the platform; external effects remain separate.
Build from primitives