Design ad click aggregation

Every layer can duplicate: the browser can follow a click twice, the redirect service can retry a Kafka produce, and Flink replays from its last checkpoint after a crash. Exactly-once is built from parts. Each click gets a unique id at the redirect (derived from the signed impression plus a nonce), and Flink deduplicates on it with keyed state that expires after an hour. Flink's checkpoints snapshot both the Kafka offsets and the operator state (window counts, dedup sets) atomically, so after a failure it rewinds both together and never counts a click twice inside the job. The sink is the last risk: if Flink wrote 'ad 42 +17 clicks' as an increment and then replayed, the database would get +34. So the sink writes absolute values keyed by (ad_id, minute, dims), an idempotent upsert, or uses a transactional two-phase-commit sink (Kafka transactions, read with read_committed). Be precise in the interview: Flink plus Kafka transactions give exactly-once inside the pipeline; with an idempotent sink the end result is effectively-once. Raw clicks in S3 make any count reproducible.

Clicks must be billed to the minute they happened, not the minute Flink processed them, or a 10-minute consumer lag shifts revenue between hours and days. Use tumbling one-minute windows on event time (the signed click timestamp, cross-checked with server receive time to stop clients faking it). A watermark of max event time minus ~30 s decides when a window is complete; Flink emits early results every 10 s for freshness and a final result at the watermark. Mobile clients and network delays still produce late clicks. Allowed lateness of a few minutes lets the window update its result (upserting the new absolute count); clicks later than that go to a side output and are counted by the batch job, which uses the full day's data with a 2-hour buffer. This is why there are two numbers: a fast real-time estimate for dashboards and pacing, and a slower exact one for billing, with automated reconciliation between them.

A big brand's IPL campaign can get 20k clicks per second on one ad. If the stream is keyed by ad_id end to end, one Flink subtask and one Kafka partition take all of it and fall behind. Two fixes. Key the Kafka topic by click_id so ingestion spreads evenly; deduplication is keyed by click_id too, so it stays spread. For aggregation, use two-stage aggregation: first aggregate by (ad_id, salt) where salt is a random 0-15, producing partial counts per minute spread over 16 subtasks, then a second stage keyed by ad_id sums the 16 partials. The second stage only sees 16 records per ad per minute, regardless of traffic. Flink's local pre-aggregation (mini-batch) achieves similar effects. The OLAP store is less affected because it receives aggregated rows, but the advertiser dashboard query for a giant campaign should hit pre-rolled campaign-level aggregates instead of summing 200 ad rows per minute.

Paying for bot clicks loses advertisers, so filtering is part of the core design. Real-time rules run in Flink: signature valid and not expired, one billable click per impression, per-IP and per-device click rate limits, datacenter IP ranges, known bot user agents, and clicks arriving impossibly fast after the impression. Heavier ML models run in the batch job, which is why billing uses batch output; real-time numbers may slightly over-count and get corrected. Budget pacing needs a different trade-off: it must be fast rather than exact. Flink adds spend to Redis per campaign; ad servers read pacing state every few seconds from a local cache. There is always some overspend from ads already shown and clicks in flight, so start throttling near 95% of budget and pace spending through the day (spend target per hour), not only stop at 100%. Overspend beyond the budget is not billed, so it is our cost; the target is to keep it under ~1%.

A streaming-only (Kappa) pipeline is simpler, but billing benefits from a recompute over complete data with better fraud models and late clicks included. Keeping the batch job as the source of record for money, and streaming for dashboards and pacing, is the standard choice.

Writing raw clicks to ClickHouse or Pinot and aggregating at query time is flexible, but at 50k/s with 5M ads and many dashboards it is expensive. Aggregating minutes in Flink cuts data ~5× and makes queries trivial, at the cost of fixed dimensions. Keep raw data in S3 for new questions.

Exact keyed state for an hour of click ids costs ~10-20 GB in RocksDB, which is fine. A Bloom filter is smaller but drops a small fraction of real clicks as false duplicates, which means under-billing. For money, exact state is worth the cost.

A server redirect guarantees we log before the user leaves, but adds a hop of latency. A beacon (navigator.sendBeacon) is faster for the user but can be lost or blocked. For billable clicks, the redirect is safer; many networks use both.

Stateless and horizontally scaled behind the CDN/LB; produce to Kafka asynchronously with batching (linger.ms ~5) and a local disk buffer if Kafka is slow, so latency stays flat.

Dedup and window state grows with traffic. Use RocksDB state backend with incremental checkpoints to S3 and state TTL; scale parallelism to the partition count.

Replaying hours of Kafka data at 5× speed needs headroom; size partitions and parallelism for replay, not only live traffic.

Redirect keeps serving 302s and buffers clicks to local disk; a sidecar ships them when Kafka returns. Users are unaffected; dashboards lag.

Restart from the last checkpoint; offsets and state rewind together, sink upserts are idempotent, so counts stay correct. Pacing state is briefly stale, so ad servers fall back to conservative throttling when data is older than 1 minute.

An alert fires when the difference exceeds 0.1%. Batch wins for billing; the team investigates (usually a dedup or late-data bug) using raw data in S3.