Design usage-based billing and metering

Kafka offers exactly-once only inside Kafka transactions; end to end you need effectively-once built from idempotency. The chain: (1) Producers create a deterministic event_id (request_id + meter), so a retry sends the same id. Random UUIDs per send defeat dedupe. (2) Ingest writes to Kafka with idempotent producers and acks=all, so an accepted event is not lost. (3) The stream processor deduplicates by event_id in keyed state with a TTL longer than the expected retry and late window (days). Flink checkpoints make state and offsets consistent; outputs go to sinks that are idempotent (upsert by aggregate key, not increment), so replaying after a failure rewrites the same values. (4) Credit burns use a unique key per customer, meter and minute, so a replay cannot burn twice. (5) The invoice run recomputes from raw events with a global dedupe over the whole period, which also catches duplicates older than the streaming TTL. The final guard is reconciliation: invoiced quantity must equal the count of unique raw events, and the streaming totals must be within a small tolerance of the batch totals. This 'stream for speed, batch for truth' structure is a lambda-style design, and it is fine here because the invoice must be exact.

Usage arrives late for normal reasons: an agent buffers during a network blip, a batch job reports storage hours once a day, or a region backlog drains hours later. Aggregate by event_time, not ingest_time, so usage lands in the correct hour and month. Flink uses watermarks to decide when a window is mostly complete, but billing aggregates should allow updates for days (allowed lateness), emitting corrected values for old buckets as upserts. The invoice run waits for a grace period after the month ends (24-48 hours) before finalising. Events that arrive after finalisation are not lost: they are rated into the next invoice as a 'prior period adjustment' line, or ignored by contract if they are older than a cut-off; either way the rule is written and visible. Corrections from producers (for example, a bug double-reported storage for a day) are new events with negative quantities or a 'void' referencing original event ids, never edits to raw data. Raw data stays immutable so you can always explain an invoice: 'these 4,213,882 events, rated with plan version 7, produced line 3'.

Limits must be fast because products check them on every request, but the exact balance comes from a pipeline that is a minute behind. Do not call the billing database on each request. Instead, the billing service computes balance and limit state per customer from per-minute aggregates and writes a small record (balance, state: ok, warn, block, updated_at) to Redis. Products keep a local cache refreshed every few seconds, so a check costs microseconds. This means a customer can overshoot by up to a minute of usage; for most products that is acceptable and should be stated in the terms. For high-value usage (expensive GPU jobs or large batch requests), use a reservation: before the job, reserve an estimated amount from the balance atomically (Redis Lua or a Postgres conditional update), and settle the actual amount after the job finishes. For very large customers, give each product node a local allowance (a slice of the remaining balance) to avoid a hot key, similar to a distributed rate limiter. Fail open with a cap: if the entitlements service is down, allow traffic for known paying customers and meter it later, because billing must not take down the product.

Rating converts quantities into money. Keep plans as versioned data, not code: each version defines meters, pricing model, tiers and currency, and is immutable once any customer uses it. A tiered price (first 1M tokens at rate A, next 9M at rate B) applies tiers to cumulative usage in the period, while a volume price applies one rate to all units based on the total, and the difference matters for invoices. Use decimal arithmetic with fixed scale, never floats, and round only at the line-item level with a documented rule. Mid-cycle plan changes split the period: usage before the change is rated with the old version, after it with the new one; fixed fees are prorated by time. Commitments ('₹10 lakh per year minimum') are evaluated at the commitment period end, with a true-up line. Credits apply in a defined order (promotional credits first, expiring soonest first). Because the rating function is pure (usage + plan version + contract → line items), you can re-rate any past period to answer disputes, test a new plan on last month's real usage before launch, and show customers an 'estimated bill' mid-month using the same code.

Streaming totals are fresh but can drift due to bugs, late data or dedupe TTL limits. Batch recompute from raw events is exact and auditable but slow. Use streaming for dashboards and limits, batch for invoices, and reconcile the two. The cost is two code paths; mitigate by sharing the rating library.

Products like Stripe Billing, Orb, Metronome or Chargebee handle rating, invoices and payments. Building in-house makes sense at large scale or with unusual pricing; metering (ingest, dedupe, aggregation) is often built in-house even when invoicing is bought. Say this in the interview: it shows judgement.

Exact blocking at zero balance needs a synchronous, consistent check per request, which adds latency and a single point of failure. Soft limits with a small overshoot are much cheaper and more reliable. Use reservations only for expensive, long-running operations.

Partitioning Kafka and Flink by customer_id puts one big customer on one partition. Partition by (customer_id, meter, hash of event_id mod N) for pre-aggregation, then combine partial sums per customer in a second stage. The balance key in Redis is split the same way with local allowances.

100k customers recomputed from S3 at once is a large scan. Pre-compute daily compacted rollups from raw events (deduped, exact) during the month so the month-end job only sums 30 days of rollups and the last day's raw events. Shard jobs and make each idempotent by (customer, period).

Producers keep events in a local disk-backed buffer (an agent or sidecar) and retry with the same event_ids. The product keeps serving. Alert if buffer age grows, because very late data delays invoices.

Dashboards show inflated usage, but invoices are recomputed from raw events, and the stream-vs-batch reconciliation flags the gap. Fix the job, replay from Kafka or S3 to rebuild aggregates (upserts overwrite), and correct credit burns with adjustment entries.

Invoice stays finalised and unpaid; dunning retries the payment on a schedule, notifies the customer, and after a grace period moves entitlements to restricted. Revenue data is unaffected because invoices are separate from payment status.