Functional
- Apply per-tenant, per-user, or per-route policies.
- Support sustained rate plus controlled burst capacity.
- Return remaining budget and retry guidance.
End-to-end walkthrough
Distributed Rate Limiter
Enforce fair request budgets across many stateless service instances.
Frame before solving
Requirements & boundaries
Quality attributes
- Decision p99 below 2 ms in-process or 10 ms over the network.
- Policy changes propagate within seconds.
- A limiter outage must not become a platform-wide outage.
Explicitly out of scope
- Billing-grade usage metering.
- Bot detection and reputation scoring.
Size the important constraints
Back-of-the-envelope estimates
A network call for every request is expensive and risky.
Counters need expiry and memory-efficient representation.
Policies can be versioned and cached aggressively.
Use local tokens or regional batching on critical paths.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/checkEvaluate principal, resource, and cost; return allow plus retry_after./v1/policies/{id}Publish a versioned rate policy./v1/usage/{principal}Inspect approximate current budget for operations.Authoritative records
policy_id, matcher, rate, burst, action, versionImmutable versions allow atomic rollout and rollback.principal+policy PK, tokens, last_refill_at, expires_atUpdate atomically with server-side time.Trace the critical path
Architecture & request flow
- 1Identify principal→
- 2Select policy→
- 3Read or increment counter→
- 4Allow or reject→
- 5Emit decision telemetry
Policy service
Own limits and rollout rules
Limiter
Make the hot-path decision
Counter store
Coordinate shared usage windows
Local fallback
Protect dependencies during store failure
Reason about the hard parts
Critical design deep dives
Algorithm boundary
Token bucket fits bursty APIs; sliding-window counters give intuitive quotas but cost more state. Use fixed windows only where boundary bursts are acceptable.
Local versus global
Lease token batches from a regional store to each gateway. The hot path spends local tokens; unused leases expire. Overshoot is bounded by lease size × active gateways.
Failure policy
Authentication, payment, and write-amplifying endpoints may fail closed; ordinary reads often fail open with a conservative local cap. Encode this choice in policy so incident behavior is predictable.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Token bucketAllows controlled burstsDistributed token state must be reconciled
Local quotasRemoves a network hopGlobal limit becomes approximate
Failure-first review
What happens if…?
Counter store is slow
Fail open or closed by endpoint risk; never use one global default.
One tenant is hot
Partition by tenant and isolate heavy hitters before they saturate shared shards.
Avoid premature complexity
How the design evolves
One service
In-process token buckets
Move here when: Single replica or best-effort limits.
Shared regional limits
Atomic counter store with cached policies
Move here when: Multiple replicas need coordinated fairness.
Fleet scale
Leased token batches, tenant isolation, regional policy plane
Move here when: Limiter latency or store QPS dominates.
Test the reasoning
Interview follow-ups
How much can a leased-token design overshoot?
Strong answer signal: Derive the bound from lease size and concurrent holders.
What clock should refill use?
Strong answer signal: Use authoritative monotonic/server-side elapsed time, not client wall clocks.
How do policy rollouts avoid a thundering herd?
Strong answer signal: Versioned push, jittered refresh, and last-known-good fallback.
Build from primitives