Design a distributed tracing system

Head-based sampling decides at the start of the request: the root service flips a coin (say 1%) and writes the decision into the sampled flag of traceparent; every downstream service obeys it. It is cheap and consistent (whole traces are kept or dropped), but it is blind: the decision is made before you know whether the request will fail or be slow, so 99% of interesting error traces are thrown away. Tail-based sampling decides after the trace finishes. All spans are sent to a Collector tier, buffered per trace_id for a decision window (10-30 s), then policies run: keep if any span has an error, keep if root duration > 2 s, keep 100% for tenant X under investigation, keep 2% of the rest. The cost is real: you must ship 100% of spans off the hosts, hold them in memory, and route every span of a trace to the same Collector with consistent hashing on trace_id. Late spans (a Kafka consumer that runs 5 minutes later) arrive after the decision; keep a short decision cache (trace_id → kept/dropped) so late spans follow the same decision. A common hybrid is head sampling at ~10-20% to cut volume, then tail sampling on that. Metrics must be computed before sampling, or your error rates will be wrong.

Tracing only works if every hop forwards the context. Over HTTP and gRPC, OpenTelemetry injects the W3C traceparent header (version, trace_id, parent span_id, flags) and optional tracestate and baggage. Auto-instrumentation covers common clients, but gaps appear in custom thread pools, goroutines started without passing context.Context, and in-house RPC frameworks; a single missing hop splits the trace in two. Async systems need explicit rules. For Kafka, the producer injects traceparent into record headers and the consumer extracts it. If one consumer poll processes 500 messages from 500 different traces, do not pick one parent: create a batch span with span links to each producer context. For long workflows (Temporal, cron-driven retries) that run for hours, putting every step in one trace creates huge, never-ending traces; better to start a new trace per step and link to the originating one. Baggage (e.g. tenant_id) is propagated to all downstream services, so keep it small and never put PII in it, because it travels to third parties too unless stripped at the edge. Also strip or regenerate incoming traceparent from external clients to stop abuse.

The main read is 'give me trace T'. Older systems (Zipkin, early Jaeger) used Cassandra or Elasticsearch with a row per span and several secondary indexes for search; the indexes cost more than the data and Elasticsearch clusters needed constant care at millions of spans per second. Tempo-style storage avoids most indexing: ingesters group spans by trace_id and write large blocks (e.g. Parquet, ~100s of MB) to S3, each with a bloom filter over the trace_ids it contains. A lookup lists blocks for the time range, checks bloom filters (false positive ~1%), and reads only matching row groups. Object storage makes 14 days of traces cheap. Search by attributes uses the columnar layout: dedicated columns for common attributes (service, http.route, status, tenant_id) are scanned in parallel by query workers, which is slower than an index but fine for 'last hour' searches. The compactor merges small blocks so lookups touch fewer files. The best entry point to traces is usually not search at all but exemplars from metrics and trace_ids in logs.

With AI features, the cause of a bad answer is usually the context given to the model: wrong retrieved chunks, a truncated prompt, a stale cached answer. So traces should include spans for retrieval (query, top-k chunk ids, scores) and for each LLM call, using OpenTelemetry GenAI semantic conventions for model name, input and output token counts and finish reason, plus cost and cache-hit flags as attributes. Full prompts and completions are large (often 5-50 KB) and contain personal data, so they should not sit in the main span store with broad access. Store them in a separate, access-controlled store (an LLM observability tool such as Langfuse, or S3 with strict IAM) linked by trace_id and span_id, after PII redaction or tokenisation at the gateway. This lets an engineer open a trace, see that the answer used chunk 7 from an outdated document, and replay the exact prompt against another model.

Head sampling is almost free and fine for small systems or when you only want rough latency breakdowns. Tail sampling keeps the traces that matter (errors, slow) but needs a stateful Collector tier with tens of GB of memory and trace-id-aware load balancing. At 200k req/s with on-call engineers who need error traces, tail sampling is worth it; use head sampling first to cap the volume entering it.

Indexed stores give fast arbitrary search but cost much more to run at millions of spans per second. Block storage in S3 is cheap and simple; search is slower and relies on columnar scans. Most teams accept slower search because they enter traces through metrics exemplars and logs anyway.

Vendor agents are quick to adopt but lock you in. OpenTelemetry SDKs and Collectors are vendor-neutral, so you can switch backends (Jaeger, Tempo, a SaaS) by changing exporter config. The cost is more moving parts you run yourself.

At 10x traffic, buffering 10-30 s of all spans needs hundreds of GB. Add head sampling for high-volume, low-value routes (health checks dropped entirely, static assets at 1%), shorten the decision window, and scale the Collector tier horizontally with consistent hashing so a resize moves only a fraction of traces.

A batch job or fan-out request can produce 100,000 spans and break the UI and the ingester. Cap spans per trace (e.g. 10k) in the Collector, and model batch jobs as many linked traces instead of one.

If the metrics generator turns user_id or full URLs into metric labels, the metrics store explodes. Use route templates (/orders/{id}) and an allow-list of dimensions.

SDKs use bounded in-memory queues and drop spans when full; agents buffer briefly and then drop. The application is never blocked. You lose traces for that window, which is acceptable; span-derived metrics alert you that spans are being dropped.

Spans buffered in it (a few seconds of a slice of traffic) are lost and some traces become partial. The load balancer reroutes its hash range to the remaining nodes. Partial traces are marked so the UI shows a 'missing spans' warning instead of a misleading waterfall.

Child spans can appear to start before their parent. The UI adjusts using the parent-child relationship (shift child spans to fit inside the parent), and hosts run NTP/chrony; durations within one span are measured with a monotonic clock, so they are still correct.