Observability: logs, metrics and traces
In one line
Metrics tell you that something is wrong, traces tell you where, and logs tell you why; design them together, with shared IDs, so you can answer questions you did not plan for.
What it is
Observability is how well you can understand what a running system is doing from the data it emits, without shipping new code. Monitoring answers known questions ('is CPU above 80%?'). Observability lets you ask new ones during an incident ('which tenant, app version and region are the slow requests coming from?').
It rests on three main signals. Metrics are numbers aggregated over time (request rate, error rate, latency histograms, queue depth); they are cheap to store and good for dashboards and alerts. Logs are timestamped records of individual events, best written as structured JSON with fields such as request_id and user_id; they hold detail but are expensive at scale. Traces follow one request across services as a tree of spans, each with a start time, duration and attributes; they show where time went in a call chain.
The signals are most useful when connected. The same trace ID appears in the logs of every service a request touched, and metrics can carry exemplars (links to a sample trace). OpenTelemetry has become the vendor-neutral standard for producing all three, and the W3C traceparent header carries the trace context between services.
Why interviewers ask
Interviewers increasingly end a design round with 'how would you know this is working in production?' A strong answer names the golden signals per service, explains latency percentiles from histograms, describes distributed tracing with context propagation, and shows awareness of cost: cardinality limits, sampling and log retention. In microservice and AI systems, being able to explain how you debug a slow request across ten services is a senior-level skill.
How it works
- 1Instrument every service with the same library (OpenTelemetry SDK or auto-instrumentation) so HTTP, gRPC, database and queue calls produce spans and metrics without hand-written code.
- 2Propagate context: the first service creates a trace ID; every outbound call adds the traceparent header; every queue message carries it in headers, so async consumers continue the same trace.
- 3Emit metrics using RED for request-driven services (Rate, Errors, Duration) and USE for resources (Utilisation, Saturation, Errors). Google's four golden signals are latency, traffic, errors and saturation.
- 4Record latency as histograms, not averages, so you can compute p50, p95 and p99 across many instances. Averages of percentiles from different servers are mathematically wrong.
- 5Keep metric label cardinality bounded: labels like route, status code, region and tenant tier are fine; user_id, order_id or full URLs create millions of time series and can take down a Prometheus server.
- 6Write structured logs (JSON) with trace_id, span_id, service, version and a stable event name. Log at the right level, and never log secrets, card numbers, OTPs or Aadhaar numbers.
- 7Sample traces. Head sampling decides at the start (for example keep 1%) and is cheap. Tail sampling decides after the trace finishes and can keep all errors and slow requests, but needs a collector that buffers spans.
- 8Ship everything through a collector pipeline (OpenTelemetry Collector, Fluent Bit, Vector) to storage: Prometheus or Mimir or M3 for metrics, Loki or Elasticsearch or ClickHouse for logs, Jaeger or Tempo for traces, or one commercial vendor.
- 9Build dashboards per service and per user journey, and alert on SLO burn rate and on clear saturation signals, not on every metric.
- 10Control cost: tier log retention (7 days hot, 30-90 days cheap storage), drop debug logs in production, and set per-team ingestion budgets.
Worked example · Debugging slow checkouts with the three signals
- Alert: the checkout latency SLO (95% under 800 ms) is burning at 10x. The dashboard shows checkout p99 jumped from 600 ms to 3.5 s at 20:05, error rate still normal.
- Metrics breakdown by label: only region=ap-south-1 and app_version=5.2 are slow; other versions are normal. Traffic is 6,000 requests per second, so the problem affects about 1,200/s.
- Click an exemplar on the latency histogram to open a sample slow trace: 3.4 s total, of which 3.1 s is a span 'pricing-service GET /offers', which itself contains 40 sequential Redis GET spans of about 75 ms each.
- Search logs by that trace_id: pricing-service logs 'offer_lookup' events showing a new code path for app_version 5.2 that loops over cart items and fetches each offer separately.
- Root cause: an N+1 query pattern against Redis, made worse because the Redis cluster's network latency rose under festive load.
- Fix: replace 40 GETs with one MGET (one round trip), deploy, and watch p99 return to 650 ms within 10 minutes. Total time to diagnose: about 15 minutes, without adding a single new log line.
Picture it
Running a hospital. Metrics are the vital-signs monitors at every bed: heart rate, blood pressure, an alarm when they cross a line. Traces are the patient's journey record: reception, lab, X-ray, doctor, with how long each step took. Logs are the doctor's detailed notes. The patient ID on all three is what lets you connect an alarm to the exact note that explains it.
Key patterns
Correlation ID everywhere
When: Any system with more than one service or any async step
How: Use the trace ID as the request ID; include it in every log line, error response and queue message so support and engineers can search one ID end to end.
Tail-based sampling
When: High traffic where keeping all traces is too expensive but errors must not be lost
How: Buffer spans in a collector until the trace completes, then keep all errors and slow traces plus a small random sample.
Exemplars
When: Latency dashboards
How: Attach a sample trace ID to histogram buckets so an engineer can jump from a slow p99 point directly to a real slow trace.
Wide structured events
When: Debugging unknown problems in complex systems
How: Emit one rich event per request with many fields (route, tenant, version, cache_hit, db_time, tokens), then slice by any field in a columnar store like ClickHouse.
Real-world use
Google Dapper
Google's 2010 Dapper paper described low-overhead distributed tracing with sampling across its services. It inspired Twitter's Zipkin, Uber's Jaeger and, eventually, the OpenTelemetry standard.
Uber Jaeger and M3
Uber built Jaeger for tracing thousands of microservices and M3 as a large-scale metrics platform, both open source, because off-the-shelf tools could not handle its volume and cardinality.
OpenTelemetry
A CNCF project that merged OpenTracing and OpenCensus. It defines APIs, SDKs and the OTLP protocol for traces, metrics and logs, so teams can switch backends (Jaeger, Tempo, Datadog, New Relic, Grafana Cloud) without re-instrumenting code.
Prometheus and Grafana
Prometheus pulls metrics from services and stores them as time series with labels; Grafana dashboards sit on top. Both are standard in Kubernetes setups at many Indian startups and product companies.
LLM observability
For AI features, each LLM call is a span with attributes such as model, prompt and completion token counts, cost, cache hit and time to first token, so teams can find which feature or tenant drives latency and spend.
In a project
A fintech startup has 25 microservices on Kubernetes, logs printed as plain text, a few CPU dashboards, and incidents that take hours to diagnose because nobody can follow a request from the mobile app through the services and the Kafka topics.
- Adopt OpenTelemetry auto-instrumentation in all services and run the OpenTelemetry Collector as a gateway in the cluster.
- Propagate traceparent over HTTP and gRPC, and in Kafka message headers so consumers continue the trace.
- Switch logs to JSON with trace_id, service, version and tenant_tier; add a redaction step in the collector for PAN, card numbers and phone numbers.
- Define RED metrics with latency histograms for every endpoint, and USE metrics for databases, Kafka consumer lag and connection pools.
- Enable tail sampling: keep 100% of error and slow traces (over 1 s) and 2% of the rest.
- Create one dashboard per critical journey (sign-up, KYC, payment) and replace static threshold alerts with SLO burn-rate alerts.
- Set retention and ingestion budgets per team, and review the top 10 noisiest log events monthly.
Impact: Mean time to diagnose incidents falls from hours to minutes because any alert leads to a trace and then to the exact logs. Observability cost stays predictable because high-cardinality labels and debug logs are controlled.
Trade-offs
Use it when
- Microservices or any request that crosses process boundaries or queues
- Systems with SLOs that need reliable latency and error measurements
- AI features where cost and latency per call vary widely and must be attributed
- Teams that run their own on-call and need fast diagnosis
Avoid it when
- Logging everything at debug level in production: storage cost can exceed compute cost
- Using unbounded labels (user_id, URL with IDs) on metrics
- Tracing 100% of traffic at high QPS without a sampling plan
- Building a custom tracing system when OpenTelemetry and an existing backend would do
Common mistakes
- Alerting on average latency, which hides the slow tail that users feel.
- Averaging p99 values from different servers instead of merging histograms.
- Losing the trace at the Kafka boundary because message headers do not carry context.
- Putting user IDs or order IDs as metric labels and causing a cardinality explosion.
- Unstructured text logs that cannot be filtered by field.
- Logging personal data, tokens or card numbers, creating a security and compliance problem.
- Having dashboards per server instead of per user journey and service.
Interview questions
Quick revision
- Metrics = what and how much; traces = where; logs = why.
- RED for services, USE for resources, golden signals: latency, traffic, errors, saturation.
- Latency as histograms; never average percentiles.
- Bound metric cardinality; IDs go in logs and traces, not labels.
- Structured JSON logs with trace_id; redact personal data.
- W3C traceparent over HTTP/gRPC and in message headers.
- Head sampling is cheap; tail sampling keeps errors and slow traces.
- OpenTelemetry for instrumentation; choose backends separately.
- Exemplars link metrics to traces.
- Business metrics (orders/min) catch problems technical metrics miss.
- Retention tiers and ingestion budgets control cost.
Check yourself
1. Which signal is best for finding which service in a 10-hop request chain added most of the latency?
2. Why is adding user_id as a Prometheus metric label a bad idea at scale?
3. Ten servers each report a p99 latency. How should you get the overall p99?
4. What does tail-based sampling allow that head-based sampling does not?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.