LLM tracing, evaluation and quality
In one line
Because LLM output is non-deterministic and can be wrong while every system metric looks healthy, you need traces that capture the exact prompt, context and response of each call, and evaluations that score quality on a fixed test set before release and on sampled live traffic after.
What it is
Normal monitoring tells you whether a service is up and fast. With an LLM, the service can be up, fast and returning 200 while giving wrong, unsafe or unhelpful answers. Quality becomes a production metric, and you need two new tools to see it.
LLM tracing extends distributed tracing: for each user request you record a span per step (retrieval, reranking, each model call, each tool call) with the model and version, prompt template version, the exact rendered prompt, retrieved chunk ids, the response, token counts, cost, latency and cache hits. OpenTelemetry has GenAI semantic conventions (attributes such as gen_ai.request.model and gen_ai.usage.input_tokens) so tools understand these spans; LLM observability tools such as Langfuse, Arize Phoenix and LangSmith store the full prompts and responses.
Evaluation (evals) measures quality. Offline evals run a fixed dataset of inputs with expected outcomes against a candidate prompt, model or retrieval change before release, like unit and regression tests. Online evals score a sample of real production traffic continuously, using user feedback, business signals and automated graders, including another LLM acting as a judge.
Why interviewers ask
When interviewers ask 'how do you know your AI feature works?' or 'the provider updated the model and answers got worse, how would you notice?', they want a concrete loop: traced requests, a versioned eval set, automated graders calibrated against humans, a release gate, and online monitoring of quality signals. It separates engineers who have run LLM features in production from those who have only built demos, and it shows up directly in prompts like 'design an LLM evaluation and observability platform'.
How it works
- 1Instrument every step with OpenTelemetry: one trace per user request, spans for retrieval, rerank, each LLM call and each tool call, and trace context propagated through queues (in Kafka message headers) so asynchronous steps join the same trace.
- 2Record LLM span attributes: provider, model and exact version, prompt template id and version, temperature, input, cached and output tokens, cost, TTFT, total latency, finish reason, cache hit, fallback used. Send full prompt and response text to an LLM observability store, after PII redaction, linked by trace id.
- 3Keep metrics low-cardinality: Prometheus metrics by model, feature and status, never by user id or prompt text. Per-tenant cost and per-prompt analysis belong in an analytics store or the LLM observability tool.
- 4Sample sensibly: keep 100% of errors, slow requests and thumbs-down conversations, and a percentage of the rest (tail-based sampling in the OTel Collector). Full prompt storage is expensive and sensitive, so set retention limits.
- 5Build the offline eval set from real traffic: 100 to 500 representative cases per feature, including hard and adversarial ones and past failures, each with an expected answer, required facts, or a rubric. Version it like code.
- 6Choose graders from cheapest and most reliable to least: exact checks (valid JSON, schema, required fields, correct label), code checks (citations point to provided chunks, no banned phrases), retrieval metrics (recall@k against labelled chunks), then LLM-as-judge with a written rubric for qualities like faithfulness and helpfulness.
- 7Calibrate LLM judges: have humans label 100 to 200 cases, measure how often the judge agrees, and fix the rubric until agreement is high. Known judge biases include preferring longer answers, preferring the first option in pairwise comparisons, and favouring outputs from their own model family; randomise order and use a different model where possible.
- 8Gate releases: every prompt, model, chunking or retrieval change runs the eval suite in CI and shows a diff against the current version per metric and per slice (language, customer tier, intent). Block on regressions beyond an agreed tolerance.
- 9Monitor online: thumbs up/down, regenerate and copy rates, escalation to a human, task success (ticket resolved, booking completed), guardrail trigger rates, and an LLM judge scoring a daily sample. Alert on drops, and pin model versions so the provider cannot change behaviour under you without a deliberate upgrade.
- 10Close the loop: when a bad answer is reported, open its trace, replay the exact prompt against other prompts or models, fix, and add the case to the eval set so it never regresses.
Worked example · Shipping a cheaper model for ticket summaries
- Goal: replace the large model used for support ticket summaries with a smaller model that costs about one fifth as much per token.
- Eval set: 400 real tickets (English, Hindi, Tamil; short and long; angry customers; tickets with refunds), each with key facts labelled by support leads.
- Graders: code check that the output is valid JSON with fields issue, customer_ask, next_step; fact recall (share of labelled key facts present, judged per fact by an LLM); LLM judge for 'contains anything not in the ticket' (hallucination), calibrated at 92% agreement with human labels on 150 cases.
- Results, current vs candidate: JSON validity 99.8% vs 99.5%; fact recall 91% vs 88%; hallucination rate 2.0% vs 2.6%. Overall acceptable within the agreed tolerance of 3 points.
- Slice view shows the problem: on Tamil tickets fact recall falls from 89% to 74%. Averages hid it because Tamil is only 8% of the set.
- Decision: route Tamil tickets (detected by a language id step in the gateway) to the old model, everything else to the new one. Blended cost per summary falls by about 70%.
- Online, a daily judge sample of 500 summaries and the agents' 'summary was wrong' button are tracked per language; after two weeks both are flat, so the change is kept.
Picture it
Tracing is the flight data recorder: when something goes wrong you can replay exactly what the pilot saw and did. Evals are the simulator checks a pilot must pass before flying a new aircraft type, plus random check rides afterwards. You need both: the recorder explains incidents, the simulator prevents many of them.
Key patterns
Trace-to-eval loop
When: Every reported bad answer
How: Open the trace, replay the exact prompt, fix the cause, and add the case to the versioned eval set as a regression test.
Cheap graders first
When: Building an eval suite
How: Use deterministic checks (schema, labels, citations, retrieval recall) wherever possible; reserve LLM judges for subjective qualities and calibrate them against human labels.
Slice-level release gates
When: Any prompt, model or retrieval change
How: Compare metrics per language, intent and customer tier, not just overall averages, and block regressions above a tolerance.
Pin and upgrade deliberately
When: Using hosted models
How: Reference dated model snapshots, run evals before switching, and roll out gradually with online monitoring.
Real-world use
OpenTelemetry GenAI semantic conventions
OpenTelemetry defines standard attribute names for model calls (model, operation, token usage and more), so traces from different SDKs and gateways can be analysed by the same tools.
Langfuse
An open-source, self-hostable LLM engineering platform that stores traces with prompts and completions, manages prompt versions and datasets, and runs scores, including LLM-as-judge evaluations. It can ingest OpenTelemetry data.
DoorDash LLM judge and guardrail
DoorDash described evaluating its support chatbot with an LLM judge across several quality dimensions and running a guardrail model online to catch ungrounded answers before they reach users.
Ragas and OpenAI Evals
Open-source frameworks for running evaluations: Ragas focuses on RAG metrics such as faithfulness and context precision and recall, while OpenAI's Evals framework provides a registry of eval tasks and graders.
In a project
A Bengaluru legal-tech startup's contract Q&A assistant got worse after a routine change, and the team only learned from a customer's escalation two weeks later. They do not know which change caused it: a new chunk size, a prompt edit or a provider model update.
- Add OpenTelemetry spans for retrieval and each model call with prompt version, model version and retrieved chunk ids; send redacted prompts and responses to a self-hosted Langfuse in the same region as the data.
- Pin the model to a dated snapshot version in the gateway config so upgrades become explicit pull requests.
- Create a 250-question eval set from real anonymised questions, with lawyers labelling the clauses that must be cited and the correct answer.
- Score retrieval recall@8 by code and answer correctness and faithfulness with an LLM judge calibrated against 120 lawyer-labelled cases.
- Run the suite in CI for every change to prompts, chunking or model config and post a per-slice diff on the pull request; block merges that drop recall or correctness by more than 2 points.
- Replay the evals on the last month's three changes: the chunk size change dropped recall@8 from 88% to 76% on long contracts. Revert it and add the failing cases to the set.
- Online: track thumbs-down rate and 'show me the clause' clicks per customer; sample 2% of traffic daily for judge scoring with an alert on a 3-point drop.
Impact: The root cause was found in an afternoon instead of weeks, later regressions are caught in pull requests, and the team can show enterprise customers a quality report with numbers, which helped close two renewals.
Trade-offs
Use it when
- Any LLM feature that real users depend on
- Before changing prompts, models, providers, chunking or retrieval
- Debugging individual bad answers reported by users
- Proving quality to enterprise customers or auditors
Avoid it when
- Storing full prompts and responses with personal data in a third-party tool without redaction, contracts and retention limits
- Using an uncalibrated LLM judge score as if it were ground truth
- Putting prompt text or user ids into metric labels, which explodes time-series cardinality
- Building a huge eval set before the feature's behaviour is settled; start with 50 good cases and grow
Common mistakes
- Monitoring only latency and errors, so quality regressions go unnoticed until customers complain.
- Not recording the exact prompt, retrieved context and model version, so bad answers cannot be reproduced.
- Breaking the trace at a queue because context was not propagated in message headers.
- Using floating model aliases that the provider updates silently.
- Judging quality by reading a handful of examples ('looks good') instead of a versioned eval set with numbers.
- Trusting an LLM judge without measuring its agreement with humans, or ignoring its biases toward long answers and its own model family.
- Looking only at averages, which hide large regressions in smaller languages or customer segments.
Interview questions
Quick revision
- Healthy metrics do not mean good answers: measure quality explicitly.
- Trace per request: spans for retrieval, rerank, model calls and tools; propagate context through queues.
- LLM span: model+version, prompt version, tokens, cost, TTFT, finish reason, cache, fallback, chunk ids.
- Full prompts go to an LLM observability store after redaction; metrics stay low-cardinality.
- Offline evals: 100-500 real cases per feature, versioned, run in CI on every change.
- Graders: exact and code checks first, retrieval recall, then calibrated LLM judges.
- Judge biases: length, position, self-preference.
- Compare per slice (language, intent, tier), not only averages.
- Online: thumbs, regenerate, escalation, task success, sampled judge scores.
- Pin model snapshots; upgrades are explicit and evaluated.
- Every bad answer becomes a new eval case.
Check yourself
1. Your AI feature shows 99.9% success and normal latency, yet complaints about wrong answers are rising. What is missing?
2. Which of these should NOT be a Prometheus metric label?
3. An LLM judge rates answers. What must you do before trusting its scores for release decisions?
4. Overall eval scores for a new model are slightly better, but Hindi questions are 15 points worse. What does this show?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.