Functional
- Ingest timestamped metric samples with labels.
- Query ranges with filtering and aggregation.
- Evaluate recording and alerting rules.
End-to-end walkthrough
Metrics Platform
Ingest, aggregate, retain, and query high-cardinality time-series data.
Frame before solving
Requirements & boundaries
Quality attributes
- Sustain bursty telemetry without blocking applications.
- Recent data queryable within seconds.
- Control cardinality and noisy tenants explicitly.
Explicitly out of scope
- Raw log search.
- Strong transactions across time series.
Size the important constraints
Back-of-the-envelope estimates
~300 MB/s before replication and index overhead.
Head index memory and churn are primary constraints.
Use object storage and compaction tiers.
Plan budgets, partial responses, and pre-aggregation.
These are reference assumptions, not universal facts. In an interview or architecture review, change them when the product context changes.
Define the contract
API & data model
Core operations
/v1/writeBatch ingest labeled samples with tenant identity./v1/query_rangeExecute bounded range expression with step and deadline./v1/rulesPublish versioned recording or alert rules.Authoritative records
tenant+fingerprint, canonical_labels, chunks, min/max timeFingerprint routes all samples for a series consistently.time_range, series_index, compressed_chunks, checksumImmutable blocks support cheap object storage and compaction.Trace the critical path
Architecture & request flow
- 1Receive samples→
- 2Validate and shard→
- 3Buffer in a log→
- 4Compact into time blocks→
- 5Query and aggregate
Ingest gateways
Validate labels and control cardinality
Durable log
Absorb bursts and enable replay
Time-series store
Compress and retain metric blocks
Query engine
Fan out and merge time ranges
Reason about the hard parts
Critical design deep dives
Ingestion path
Gateways validate labels, enforce per-tenant series/sample budgets, and shard by series fingerprint. A replicated log absorbs bursts; ingesters build compressed in-memory chunks and periodically flush immutable blocks.
Cardinality control
Limit new-series creation, label lengths, and per-metric dimension counts. Expose rejected-label diagnostics. Heavy-hitter detection identifies accidental user IDs or request paths before memory collapses.
Query execution
A frontend splits ranges, deduplicates requests, caches stable blocks, and enforces byte/series/concurrency budgets. Queriers push filters to blocks and merge partial aggregates rather than raw samples when possible.
Make trade-offs explicit
Architecture decisions
ChoiceWhyCost
Shard by seriesKeeps one series orderedHigh-cardinality tenants can skew shards
DownsamplingControls long-term costRemoves fine-grained historical detail
Failure-first review
What happens if…?
Cardinality explodes
Enforce tenant budgets and drop low-value dimensions before storage.
Query fans out too widely
Limit concurrency, cache rollups, and require narrower time ranges.
Avoid premature complexity
How the design evolves
Single cluster
In-memory head + local block store
Move here when: Team-scale observability.
Durable scale
Write log, sharded ingesters, object blocks
Move here when: Ingestion exceeds one node and retention grows.
Platform
Tenant budgets, query scheduler, downsampling, regional cells
Move here when: Noisy neighbors and global reliability matter.
Test the reasoning
Interview follow-ups
Why shard by series instead of time?
Strong answer signal: Keep one series ordered/compressible while time blocks serve retention.
What breaks first during a cardinality explosion?
Strong answer signal: Head index memory, metadata churn, and compaction—not raw disk alone.
Can alerts tolerate partial query results?
Strong answer signal: Usually fail-safe with explicit missing-data semantics; never silently treat partial as complete.
Build from primitives