Functional
- Publish messages to named queues or partitions.
- Lease work to consumers and acknowledge completion.
- Retry with delay and quarantine poison messages.
End-to-end walkthrough
Distributed Queue
Buffer work durably while coordinating ownership, retries, and poison messages.
Frame before solving
Requirements & boundaries
Quality attributes
- Durable acknowledgement survives broker loss.
- Ordering is preserved within the documented scope.
- Backlog growth is observable and bounded by retention.
Explicitly out of scope
- Arbitrary stream joins and analytics.
- Exactly-once external side effects.
Size the important constraints
Back-of-the-envelope estimates
Batch sequential appends across many partitions.
Large payloads should use object references.
Raw storage is roughly 1.2 PB before replication at peak.
Coordination must avoid per-message central locks.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/queues/{queue}/messagesPublish payload, key, delay, and deduplication token./v1/queues/{queue}/leasesFetch a bounded batch with visibility timeout./v1/messages/{id}/ackAcknowledge or negatively acknowledge a lease generation.Authoritative records
partition+offset, id, key, payload_ref, available_at, attemptOffset is immutable; delivery state is separate.message_id, consumer, generation, expires_atGeneration fences late acknowledgements from previous owners.Trace the critical path
Architecture & request flow
- 1Append message→
- 2Assign partition→
- 3Lease to consumer→
- 4Acknowledge success→
- 5Retry or dead-letter failure
Brokers
Persist partitioned message logs
Coordinator
Assign consumers to partitions
Consumers
Process and acknowledge work
Dead-letter store
Quarantine repeatedly failing messages
Reason about the hard parts
Critical design deep dives
Durability boundary
A broker appends to its local log and replicates to a quorum before acknowledging. Producers retry with a stable producer/message sequence so uncertain acknowledgements do not create logical duplicates.
Lease semantics
The visibility timeout must exceed normal processing but remain recoverable. Consumers heartbeat long jobs. A lease generation prevents an expired worker from acknowledging work now owned elsewhere.
Backpressure and fairness
Track oldest-message age per tenant and queue. Enforce publish quotas, weighted consumer scheduling, maximum retry rates, and dead-letter thresholds so one poison workload cannot starve others.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Partition orderingScales beyond a single sequenceNo total order across partitions
Leased deliveryRecovers abandoned workSlow jobs can be delivered twice
Failure-first review
What happens if…?
Consumer dies mid-task
Let the lease expire and redeliver; handlers must be idempotent.
Poison message loops
Cap attempts and preserve payload plus failure context in a dead-letter queue.
Avoid premature complexity
How the design evolves
Work queue
Single replicated log and leased consumers
Move here when: Basic background jobs.
Partitioned service
Keyed partitions and consumer groups
Move here when: Throughput or local ordering grows.
Multi-tenant platform
Quota isolation, tiered storage, delay index
Move here when: Retention, fairness, and large delays dominate.
Test the reasoning
Interview follow-ups
Can this queue guarantee exactly-once processing?
Strong answer signal: Only within controlled state transitions; external effects need idempotency/transactions.
How do delayed messages avoid scanning the log?
Strong answer signal: Time-bucketed delay index promotes due messages.
Which metric detects consumer trouble earliest?
Strong answer signal: Oldest message age, processing latency, and retry rate—not depth alone.
Build from primitives