Design centralised logging

There are two families of log stores. Elasticsearch/OpenSearch builds an inverted index over every word in every line. Any free-text search is fast, but the index is often as large as the raw data, it lives on SSD, and indexing burns CPU. At 5 TB/day that means roughly 5-7 TB/day of hot SSD storage plus replicas, which is very expensive. The other family (Grafana Loki, and also columnar designs on ClickHouse) indexes only a few low-cardinality labels such as service, namespace, level and region. The log text is stored as compressed chunks in S3. A query first selects streams by label, then brute-force scans those chunks in parallel. Storage is 5-10x cheaper and ingestion is simple; the cost moves to query time, which is fine because 95% of queries are 'this service, this hour, contains X'. ClickHouse sits between the two: columnar storage with good compression, fast scans, and optional token bloom-filter skip indexes for text. A good answer picks label-indexed or columnar storage for the bulk of logs, and keeps full-text indexing only for small, high-value streams (for example security logs) where needle-in-haystack search across all services matters.

Log volume is highest exactly when the system is failing: services log every retry and stack trace, and engineers run many queries at once. A logging pipeline that falls over during an outage is worse than useless. The design defends in layers. First, the agent has a bounded memory buffer plus a bounded disk buffer; when both fill, it drops by priority (debug first, then info, keep warn and error) and counts what it dropped, instead of blocking the application's stdout. Second, the distributor enforces per-team ingest rates with a token bucket and returns 429, so one team's crash loop logging 50 MB/s cannot starve everyone. Third, Kafka sits between ingestion and indexing with hours of retention, so if ingesters or S3 are slow, lag grows but nothing is lost; ingesters catch up later. Fourth, on the read side, the query frontend splits and queues queries per tenant with limits on bytes scanned and concurrency, so one engineer's 30-day regex over all services does not block the on-call queries. Finally, services should rate-limit their own repeated logs (log the first 10 identical errors per minute, then a counter).

The biggest operational mistake in label-indexed stores is high-cardinality labels. If a team adds user_id as a label, every user creates a new stream: millions of tiny chunks, a huge index and slow queries. The distributor should reject labels outside an allow-list and cap streams per tenant (e.g. 10,000 active streams). High-cardinality values belong inside the line as structured fields and are filtered at query time (| json | order_id="123"). Structured JSON logs with consistent field names (ts, level, service, trace_id, msg) make parsing cheap and enable jumping to traces. Cost control is a product feature, not an afterthought: show each team its GB/day and cost in a dashboard, make debug logging off by default in production, sample noisy success logs (keep 1% of 200 OK access logs, 100% of errors), and set retention by class. Many companies find 20-30% of log volume comes from a handful of noisy services; fixing them is cheaper than any storage optimisation.

Agents can push straight to ingesters (simpler, fewer parts, lower delay). Kafka adds cost and one more system to run, but gives hours of buffering, replay after an indexer bug, and the ability to feed other consumers (security SIEM, anomaly detection) from the same stream. At 5 TB/day with strict 'never lose logs during an incident' requirements, Kafka is worth it; a startup with 50 services can skip it.

Fluent Bit is small (a few MB of memory) and battle-tested as a DaemonSet. Vector has a strong transform language and good performance. The OpenTelemetry Collector unifies logs, metrics and traces and the OTLP format, which reduces vendor lock-in. Any is fine; what matters is bounded buffers, disk spooling, offset tracking and redaction at the edge.

Managed tools are fast to adopt but are usually priced per GB ingested and indexed; at terabytes per day the bill becomes one of the largest infra lines. Self-hosted Loki or ClickHouse on S3 is far cheaper per GB but needs a platform team. Data-localisation rules (RBI for payment data in India) can also force logs containing payment details to stay in Indian regions.

A single chatty service (e.g. API gateway access logs at 30 MB/s) maps to one stream and one ingester. Shard it: add a synthetic shard label (stream_shard=0..15) at the distributor for streams above a rate threshold, spreading the load; queries fan out over shards automatically.

A 30-day regex over all services can scan 20 TB. Enforce max bytes scanned per query, require at least one label selector, cache results per time split, and offer pre-computed metrics (log-derived counters by the ruler) for trend questions instead of raw scans.

Many low-volume streams flush tiny chunks, raising S3 PUT cost and query fan-out. Use a max chunk age with a minimum size, and let the compactor merge index files daily.

Agents keep buffering to local disk (bounded) and retry with backoff; if Kafka is down for longer than the buffer, they drop lowest-priority logs first and emit a 'logs dropped' metric. If S3 is down, ingesters keep data in memory and WAL and stop committing offsets, so Kafka holds the backlog; once S3 recovers they flush and catch up.

Each ingester replays its WAL on restart. With replication factor 3 (or because offsets are committed only after flush), no acknowledged data is lost; queries for very recent data may briefly miss the crashed ingester's in-memory lines.

A bad deploy turns on debug logging everywhere and volume jumps 10x. Per-team rate limits contain it to that team; alerts fire on per-team ingest rate; the team's own logs are throttled rather than everyone's.