Design a metrics and monitoring system

Monitoring data has a special shape: each series gets samples at nearly regular intervals, values change slowly, and data is written once and read in time ranges. A TSDB exploits this. Samples are grouped per series into chunks of ~120 samples. Timestamps are stored as delta-of-delta: with a 15 s interval, most deltas-of-deltas are zero and take 1 bit. Values use Gorilla XOR encoding: XOR with the previous value, storing only the meaningful bits; slowly changing values compress to a few bits. Facebook's Gorilla paper reported ~1.37 bytes per sample, versus 16 bytes raw. Recent data lives in memory in a head block, protected by a write-ahead log; every 2 hours it becomes an immutable block on disk or object storage with its own inverted index (label=value → series IDs) and chunk files. Immutable blocks make compaction (merge small blocks into larger ones, remove duplicated replicas, apply deletions) and downsampling simple background jobs. Queries locate series via the index, then read and decode only the chunks in the time range. Updates in place do not exist, which is exactly why it is fast.

One Prometheus server handles a few million active series; beyond that you need a horizontally scaled design. Distributors are stateless: they receive writes, validate, and route each series by a hash of its tenant and labels onto a consistent hash ring of ingesters, with replication factor 3 placed across availability zones. A write succeeds when a quorum (2 of 3) acknowledges, so one slow or failed ingester does not block ingestion. Because every sample of a series goes to the same 3 ingesters, they can compress efficiently. Ingesters are stateful: when one leaves, it must flush or hand over its in-memory series; when one crashes, it replays its WAL on restart, and the other two replicas covered the gap. Replicas create three copies of each block in object storage; the compactor deduplicates them. Hot tenants are handled with shuffle sharding: each tenant uses a subset of ingesters, so one tenant's explosion affects only its subset. Scale queries independently: queriers are stateless, store gateways shard blocks among themselves, and the query frontend adds splitting, caching and per-tenant fairness queues.

Cost and memory scale with the number of active series, and each unique combination of label values is a new series. A developer who adds user_id, order_id or a full URL path as a label can create millions of series in minutes, blowing up ingester memory and slowing every query. Defend in layers. At the source: client libraries and code review rules (labels must be bounded sets; use route templates like /orders/{id}, not raw paths). At ingestion: per-tenant limits on active series, series per metric and label value length, enforced by distributors, which reject new series beyond the limit while keeping existing ones flowing. Visibility: a cardinality explorer showing top metrics and labels by series count, and alerts when a tenant's series grows 2x in an hour. Also manage churn: Kubernetes pods change names on every deploy, so pod labels create new series each deploy; short head-block retention and dropping pod labels for aggregated metrics help. Per-user or per-request data belongs in logs, traces or an OLAP store, not metrics.

Alerting is the part people depend on at 3 am, so it needs higher reliability than dashboards. The ruler evaluates rules on a schedule against the same query path, sharded by rule group with leases so each group is evaluated by exactly one ruler at a time. Use 'for: 5m' durations to avoid paging on blips, and prefer alerting on symptoms tied to SLOs (error budget burn rate over 1 hour and 6 hours) over causes like CPU. Alertmanager deduplicates (the ruler re-sends firing alerts), groups related alerts into one notification, applies silences during maintenance and inhibitions (do not page for 500 services when the whole cluster is down), and routes by team labels. Run it as a replicated cluster that shares notification state via gossip, so failover does not double-page or miss. Finally, the monitoring system cannot alert about its own death. Use a dead man's switch: an always-firing 'Watchdog' alert sent to an external service that pages if the heartbeat stops, plus a small independent Prometheus that monitors the monitoring stack itself.

Pull (scrape) gives the server control of rate, makes 'target is down' detectable (scrape fails), and is simple to debug with curl. Push suits short-lived jobs, serverless functions and clients behind NAT. Use agents that scrape locally and push (remote write) to the central system, which combines both, and OTLP push for apps instrumented with OpenTelemetry.

SaaS removes operational load, but costs scale with series and hosts, and large Indian companies often find bills in crores per year and data-residency questions. Open-source stacks (Prometheus + Mimir/Thanos/VictoriaMetrics) cost engineers' time but are cheaper at scale. Mention FinOps: cost per team based on series counts.

Keeping 15-second data for 15 months makes every long query scan billions of samples. Downsampling to 5-minute and 1-hour aggregates (min, max, sum, count) keeps long queries fast and storage small, losing fine detail that is rarely needed after a month.

A large deploy replaces thousands of pods, creating new series for every metric and making ingesters hold old and new series at once. Plan memory for ~2x active series during deploys, roll deploys in waves, and avoid high-churn labels on high-volume metrics.

A query over all services for 90 days at 15-second resolution can scan terabytes. The query frontend enforces limits (max samples, max series, time range), splits by day and caches, and auto-uses downsampled data for long ranges. Recording rules precompute expensive aggregates used by dashboards and alerts.

Writes continue with 2 of 3 replicas (quorum). The ingester restarts, replays its WAL and rejoins. If it is lost permanently, its data since the last block upload exists on the other replicas. Queries merge replicas, so users see no gap.

Ingesters keep data in memory and on local disk longer and retry uploads; recent queries still work from ingesters. Long-range queries fail or degrade. Alerting uses recent data only, so it keeps working, which is a key design choice: alert rules should not depend on historical queries.

Agents buffer samples in a local WAL for a few hours and replay when connectivity returns. Local alerting in the agent (or a small per-cluster Prometheus) covers critical alerts during network partitions.