LLM cost engineering

With classic backends, cost is mostly fixed infrastructure that grows slowly with traffic. With LLMs, every call has a direct, variable price: input tokens x input price + output tokens x output price, where output tokens typically cost 3 to 5 times more than input tokens. One customer, one feature or one runaway agent loop can cost 100 times another, so cost becomes a design property like latency.

Cost engineering has two halves. Reduction: shrink calls, tokens and price per token without losing quality. Visibility and control: attribute every call's cost to a tenant, feature and outcome, set budgets and alerts, and use unit economics (cost per resolved ticket, per generated report) to make product decisions.

In India the pressure is sharper: prices are in dollars while revenue per user is often in rupees and low, and Indian-language text can need several times more tokens than the same message in English on common tokenizers. A feature that is cheap for a US SaaS seat can be unaffordable for a free consumer app with millions of users.

1. 1Model the cost per unit of value: per request, per conversation (number of calls x tokens per call) and per successful outcome. Use the provider's reported usage fields, including cached tokens, not your own estimates.
2. 2Avoid calls: answer common, structured requests with rules, search or templates; generate on user action rather than on page load; precompute and store results that many users share (product summaries) instead of generating per view.
3. 3Shrink input tokens: trim conversation history (keep the last few turns plus a running summary), send 5 relevant chunks instead of 30, remove verbose instructions and unused tool definitions, and compress JSON context to only needed fields.
4. 4Order prompts for provider prompt caching (static first, dynamic last); cached input tokens are billed at a large discount, often 50 to 90% depending on the provider.
5. 5Shrink output tokens: set max_tokens per feature, ask for concise formats (bullets, JSON fields) and stop sequences, and let users stop generation. Output is the expensive side.
6. 6Right-size the model: route simple tasks (classification, extraction, short rewrites) to small models and keep large models for hard reasoning; use a cascade (try small, escalate on low confidence or failed validation). Prove each routing rule with eval scores per slice.
7. 7Use batch pricing for non-urgent work: OpenAI and Anthropic batch APIs process requests asynchronously within 24 hours at about half the price, ideal for nightly summaries, re-embedding and evaluations.
8. 8Consider fine-tuned small models or self-hosting (for example open-weight models on vLLM) only at high, steady volume: you pay for GPUs per hour whether busy or idle, so the break-even depends on utilisation, and you take on serving, scaling and upgrades.
9. 9Control spend: per-tenant and per-feature token budgets in the AI gateway, step and cost caps for agents, and anomaly alerts (for example spend above 3x the same hour last week).
10. 10Attribute cost: the gateway emits a usage event per call (tenant, feature, model, input, cached and output tokens, price version); store it in an analytics database and join with a versioned price table, because prices change and history must not be recomputed with today's price.