Design a cost-per-request FinOps dashboard for AI features

LLM cost looks simple (tokens × price) but is easy to get wrong by 10-30%. Use the provider-reported usage, not your own token estimate; tokenisers differ and providers bill on their count. Price each token class separately: cached input tokens are discounted heavily by most providers, some providers charge a premium for writing to the prompt cache, reasoning tokens are billed as output even though the user never sees them, and batch APIs are typically about 50% cheaper. Calls that failed after partial output, cancelled streams and retries still cost money, so record them with status and retry_of so retry waste is visible. Prices change, so the price book is versioned with effective dates and each row stores the price_version_id used; recomputing last month with today's price is a classic bug. Currency: providers bill in USD, so store the daily USD-INR rate used. Finally, contract discounts and credits apply at invoice level, so show 'list cost' per call and apply a monthly true-up factor for 'effective cost', and reconcile computed totals against the invoice every month; a gap above 2% usually means untracked traffic bypassing the gateway or a missing price class.

Cost per tenant is only as good as the tags on each call. The gateway is the choke point: all LLM calls must go through it (block direct provider keys with network policy and by not handing them out). Tags come from the original request context: tenant, feature, endpoint and root request id travel as OpenTelemetry baggage, are copied into Kafka message headers by producers, into Temporal workflow memo or headers for durable agent runs, and into batch jobs as job metadata. The common failure is the async hop: a summary generated by a background worker loses the tenant because nobody copied the header. Make this measurable: untagged calls are still recorded as 'unattributed' with the calling service, attribution coverage is a dashboard metric with an owner, and CI tests check that critical async paths carry the tags. For agents, one user request can trigger 20 LLM calls; the root_request_id ties them together so you can report cost per agent run and, by joining with outcome events (ticket resolved, draft accepted), cost per successful outcome, which is the number product managers actually need to price a feature.

LLM tokens are often the largest line, but RAG features also use vector databases, embedding jobs, Postgres, Kafka and Kubernetes compute that serve all tenants together. Allocation is an estimate, so the goal is a method that is simple, documented and stable. Step one: get cost per service. The AWS Cost and Usage Report (amortised, so reserved instances and savings plans are spread fairly) gives cost per resource and tag; OpenCost or Kubecost splits node cost across namespaces and pods, usually by the larger of requested and used CPU and memory. Step two: split each service's cost across tenants by a usage key that tracks what drives its cost: queries or CPU-seconds from traces for a stateless service, vectors stored for the vector DB, bytes stored for object storage. Step three: show what cannot be fairly allocated, such as idle capacity and platform overhead (observability, CI), as separate lines rather than smearing them across tenants, so teams see their waste. Version the allocation method so a change does not silently rewrite last quarter's margins. Billing data lags by up to a day, so infra cost is daily while LLM cost is near real-time; the dashboard labels which numbers are final.

Different problems need different detectors. Slow drift (a new prompt version is 30% longer; a big tenant onboarded) shows in daily data: compare today's cost per (tenant, feature) with the trailing 14 days for the same weekday and alert when it is more than ~3 standard deviations above the mean or more than 2× yesterday. Sudden runaways (an agent loop, a retry storm, a scraper abusing a free-tier tenant) burn money in minutes and cannot wait for a daily job. A streaming job (Flink, or ClickHouse materialised views queried every minute) keeps 5-minute spend windows per (tenant, feature) and compares them with the same hour-of-week baseline. To avoid alert fatigue, require both a relative jump and an absolute floor (a tenant going from ₹2 to ₹20 is not an incident), group alerts by owning team using a feature-to-team mapping, and attach the likely cause: changes in call count, tokens per call, model mix or prompt version. For features that opt in, the anomaly can act: push a temporary budget to the gateway so that one tenant-feature pair gets 429s while everyone else is unaffected. This turns the dashboard from a report into a control system.

Prometheus is great for service metrics but each unique label set is a time series; 5,000 tenants × 30 features × models creates hundreds of thousands of series and kills it. ClickHouse handles high-cardinality event data and arbitrary group-bys cheaply. Keep only low-cardinality totals (spend per feature) in Prometheus for alerting next to service SLOs.

Computing at ingest makes queries simple and fast and freezes the price used. Computing at query time from tokens allows re-pricing scenarios ('what if we moved to model B?'). Do both: store cost and price_version_id at ingest, and keep token counts so what-if analysis can re-price on demand.

Per-request CPU tracing can allocate infra almost exactly but costs engineering effort and its own compute. A few well-chosen usage keys per service get within a few percent, which is enough for pricing decisions. Spend precision where the money is: LLM cost exact, infra allocated.

Auto-throttling stops runaway spend fast but can hurt a paying customer during a legitimate spike. Make it opt-in per feature, scope it to one tenant-feature pair, and require an absolute floor and a human-visible reason.

Kafka absorbs bursts; ClickHouse needs larger batch inserts and more shards keyed by tenant hash. Rollups keep dashboard queries fast regardless of raw size; raw rows can be downsampled to hourly after 90 days.

Ad-hoc queries on billions of raw rows can be slow and expensive. Route standard views to materialised rollups, put query time limits on raw access, and precompute top-N tenants and features daily.

The gateway writes usage to a local disk buffer if Kafka is unavailable and replays later; event_id dedupes. A freshness alert fires if no events arrive for a feature that normally has traffic. The monthly invoice reconciliation is the final safety net.

A typo makes a model 10× more expensive. Price changes go through review, are validated against the provider's published price, and because rows store price_version_id, affected rows can be recomputed precisely after the fix.

A team calls a provider directly with its own key: the invoice is higher than computed cost. Reconciliation by API key shows the gap; prevent it with egress network policy and central key management.