Design an LLM evaluation and observability platform

The first rule of an observability platform is that it must fail silently. The SDK writes spans to a bounded in-memory buffer and flushes from a background worker; the request thread never waits on the network, never retries and never throws. If the buffer fills (platform down, network slow), it drops spans and counts the drops so we know data is incomplete. The ingest API is stateless and does almost nothing: authenticate, check size limits, write to Kafka, return. Kafka absorbs bursts (a festive sale can triple traffic) and lets redaction and storage workers fall behind for minutes without losing data, with 24-72 h retention as a safety net. Partitioning by trace_id keeps spans of one trace together, which helps late-arriving children join their parent. Writes to ClickHouse are batched into large inserts because each insert creates a data part, and thousands of tiny inserts per second cause 'too many parts' errors. Duplicates from SDK or Kafka retries are expected (at-least-once); span_id is the dedupe key and ReplacingMergeTree collapses duplicates during merges, while queries that must be exact use FINAL or argMax. For tail sampling of non-LLM spans, run it in an OpenTelemetry Collector tier: keep 100% of errors, slow traces and LLM spans, and 10% of the rest.

An LLM judge is a model too, so it has bias and noise. Known problems: position bias in pairwise comparisons (prefers the first answer), verbosity bias (prefers longer answers), self-preference (a model rates its own family's outputs higher), and drift when the judge model is upgraded. Mitigations: write narrow rubrics with concrete criteria and examples rather than 'rate quality 1-10'; prefer binary or 3-level labels (supported / partially / unsupported) which are more consistent than fine scales; for pairwise, run both orders and only count consistent wins; use a judge from a different model family than the generator where possible; pin the judge model and prompt as an evaluator_version so scores are comparable over time. Most importantly, calibrate: take 200-300 traces, have two human annotators label them, measure human-human agreement, then judge-human agreement (Cohen's kappa or simple accuracy). If the judge agrees with humans about as often as humans agree with each other, use it; if not, fix the rubric. Re-run this calibration whenever the judge model changes. Deterministic checks (schema valid, citation exists, SQL executes) should always run before the judge, because they are free and never wrong.

Engineers ask two kinds of questions. Aggregate questions: 'p95 latency and average faithfulness by prompt version for the last 7 days', 'which tenants had the most JSON parse failures'. These scan millions of rows but only a few columns, which is exactly what a columnar store like ClickHouse is good at: it reads only the needed columns, compresses repeated values like model names very well, and aggregates billions of rows in seconds. Lookup questions: 'show me the exact prompt for this trace'. These need large text blobs but only for a handful of traces. Putting 8 KB bodies into ClickHouse rows would bloat every scan and the storage bill; putting them in S3 costs about $0.025 per GB-month and is read only when someone opens a trace. The span row keeps a body_ref pointer. Full-text search over prompt bodies is a real request ('find answers mentioning a refund of more than ₹5,000'); handle it with a small token index or ngram bloom filter in ClickHouse on a truncated preview column (first 1 KB) rather than indexing all bodies in Elasticsearch, which would be the most expensive part of the system. Postgres holds the small, transactional things: prompts, datasets, experiments, annotation queues.

LLM outputs are noisy, so comparing two averages from one run is misleading: a 1-point drop on 100 examples is usually noise. Make the comparison paired: run baseline and candidate on the same items and look at per-item differences, which removes most item-difficulty variance. Use a bootstrap over items to get a 95% confidence interval for the delta, and define the gate as 'fail if the interval is entirely below -X', where X is the drop the product team agreed is acceptable. Size the dataset for the effect you want to catch: with paired items and ~10% of items changing outcome, roughly 800-900 items are needed to reliably detect a 3-point drop; 100 items only catch large regressions. Freeze everything that is not under test: dataset version, evaluator versions, judge model, temperature 0, and cache baseline outputs so each run only pays for the candidate. Slice results by tag (Hindi queries, long documents, refund questions) because an overall average can hide a large regression on one slice. Gate on cost and latency too: a prompt that is 2% better but 40% more expensive should be a conscious decision, not an accident.

OTel with the GenAI semantic conventions gives vendor-neutral spans, auto-instrumentation for HTTP, Kafka and databases, and context propagation for free, so LLM spans sit in the same trace as everything else. A thin custom SDK is still useful for LLM-specific helpers (prompt fetch with caching, scoring from code, dataset capture). Best of both: accept OTLP at ingest and ship a small SDK that wraps OTel rather than replacing it.

Full prompts make debugging possible: most bad answers come from bad context (wrong chunks, truncated history), which metadata cannot show. The cost is storage and privacy risk. Store redacted full bodies for 30 days, metadata for 13 months, and allow tenants or features handling sensitive data (health, finance) to opt into metadata-only capture.

Judging 100% of traffic can cost as much as the product's own LLM bill. Sampling 1-5% gives stable daily trends for each feature; targeted sampling (all negative-feedback traces, all traffic of a new prompt version for its first day) puts the budget where the risk is. Cheap heuristics run on 100%.

SaaS is faster to adopt, but prompts contain customer data, and Indian regulated businesses (RBI-regulated lenders, payment companies) often must keep data in India. Self-hosting an open-source core like Langfuse on our own ClickHouse, S3 and Postgres keeps data in-region and costs engineers' time to operate.

At 30k spans/s, many small inserts create too many parts and merges fall behind. Increase batch size and interval (100k rows or 5 s), use async inserts, and add shards keyed by tenant_id hash. Kafka gives the buffer while you scale.

One team's agent in a retry loop emits 50× normal spans. Enforce per-project ingest quotas at the ingest API (token bucket in Redis; return 429 which the SDK treats as 'drop and count'), so one tenant's bug cannot delay everyone else's traces.

As 50 teams add evaluators, judge spend can exceed budget. Give each project an evaluation budget, cache judge results by (evaluator_version, input hash, output hash), and push teams toward deterministic checks.

SDK buffers fill and drop spans; products are unaffected because nothing is on the request path. Prompt fetch uses the locally cached last-known-good version, so a platform outage never blocks a prompt load. When ingest recovers, Kafka consumers catch up; the dropped-span counter tells us exactly which period has gaps.

Redaction happens before bodies reach S3 or ClickHouse, so a bug means raw PII is stored. Detect it with a canary: synthetic traces containing known fake PII are sent every minute and an alert fires if they are found unredacted. Remediation is a job that re-redacts or deletes the affected S3 objects and ClickHouse rows for that time range. Keep the raw Kafka topic's retention short (24-72 h) and encrypted, because it is the one place where unredacted data lives by design.

Scores shift overnight and dashboards show a fake regression. Pin judge model versions in evaluator_version, run the old and new judge side by side on the calibration set before switching, and annotate dashboards at the switch point.