Design real-time fraud detection for payments

Fraud scoring sits inside authorisation, so every millisecond is paid by every payment. A practical budget at p99: 5 ms network and parsing, 10 ms feature reads, 5 ms model inference, 2 ms rules, leaving headroom inside a 50 ms deadline. To hit it: read all features in one pipelined batch (or one multi-get per shard) instead of 20 sequential calls; keep the rules and model in memory on the scoring nodes or in a sidecar to avoid an extra hop; use tree models that infer in microseconds to a few milliseconds on CPU; avoid any synchronous graph traversal or database query. Then decide the timeout behaviour explicitly, because a scoring outage will happen. Fail-closed (block when unsure) stops fraud but turns an outage into a payments outage. Fail-open (allow everything) keeps payments flowing but invites attackers who notice. The usual answer is a degraded policy: if the model or feature store times out, apply a simple local rule set (amount caps for new devices, block list from local cache) and allow the rest, and alert immediately. Hedged requests to a second replica after p95 latency also cut tail latency. Interviewers want to hear the budget broken down and the fallback chosen deliberately.

Most fraud signal is 'how unusual is this compared to recent behaviour': how many payments from this device in 10 minutes, how many new payers sent money to this VPA today, total amount from this card in 24 hours. These are sliding-window aggregates over entity keys. Compute them in a stream processor (Flink) keyed by entity, with windows built from small buckets (for example 60 one-minute buckets for a 1-hour window), and push the current values to the online store after each event. An alternative used by many teams is to update counters directly in Redis on the authorisation path (INCR on time-bucketed keys with TTL), which gives the freshest data but adds writes to the hot path; a hybrid keeps a few critical counters inline and the rest in Flink. Freshness matters: attackers run bursts of attempts within seconds, so a feature that lags a minute misses the attack; aim for under a second. Use the same feature definitions for training: logging the exact feature vector at decision time into the decision log is the simplest way to avoid training/serving skew, because training then uses exactly what the model saw. Watch out for hot keys: a big merchant's counters get updated thousands of times per second; shard the counter and sum on read.

Rules and models do different jobs. Rules are for known, crisp patterns and for reacting within minutes to a new attack ('block payments over ₹5,000 to VPAs created in the last 24 hours from this bank'), for regulatory requirements, and for allow lists. They are readable and fast to change but brittle and hard to maintain once there are thousands. The model (gradient-boosted trees such as XGBoost or LightGBM) generalises across hundreds of weak signals and is robust to small changes. Combine them in a decision policy layer: hard block rules first, then the model score, then thresholds that depend on context (amount, merchant category, user tenure), mapped to allow, challenge or block. Challenge (OTP, 3-D Secure, PIN re-entry, cooling-off for first-time large transfers) is valuable because it stops many fraudsters while costing honest users only a little friction. Every rule should run in shadow mode before going active, and rules need owners and expiry dates, or the rule set rots. Measure both sides: fraud basis points caught and false positive rate (good users declined or challenged), with the money impact of each, because a false decline often loses a customer, not just a transaction.

Fraud labels are hard. Chargebacks can arrive weeks after the payment; for UPI, fraud often shows up as user complaints and bank reports rather than chargebacks. Many frauds are never reported, so 'not labelled' does not mean 'legitimate'. Train on transactions old enough that most labels have arrived (for example 60-90 days), and use the time when a label became known so the training set only contains information available at that date. There is also selection bias: transactions you blocked never produce outcome labels, so the model learns only from what earlier systems let through. Counter this by letting a tiny random sample of would-be-blocked, low-amount transactions go through with step-up authentication (exploration), or by using analyst review outcomes on blocked cases. Evaluate on a time-based holdout because fraud patterns drift. Monitor in production: score distribution, block rate, challenge pass rate and feature drift per day; a sudden change in block rate usually means a broken feature, not a fraud wave. Feed analyst verdicts back quickly as fast labels, and use them to retrain more often for the patterns attackers change fastest.

Inline scoring can stop fraud before money moves, which is essential for UPI where transfers are instant and hard to reverse. It adds latency and becomes part of the payment's availability. Async detection after authorisation is cheaper and can use heavier models and graph analysis, but can only catch fraud after the fact (useful for blocking accounts and mule detection). Use both: inline for decisions, async for deeper analysis that updates features and block lists.

Flink-computed features keep the auth path read-only and support complex windows, but have a small lag. Inline Redis counters are freshest but add writes and failure modes to payments. Most teams do a hybrid with a few inline counters for burst attacks.

Trees are fast on CPU, strong on tabular features and easy to explain with SHAP. Deep models (sequence models over a user's history, graph neural networks) can catch subtler patterns but cost more to serve and explain. Start with trees; add embeddings from deep models as features computed offline.

At 500k transactions/s, feature reads reach ~10M keys/s. Co-locate features for an entity in one hash so one read fetches many features, keep a small in-process cache for slow-changing profiles (merchant risk, user tenure), and scale the in-memory store horizontally by entity key.

A large merchant during a sale produces thousands of updates per second on the same counter. Shard counters into N sub-keys and sum on read, or have Flink pre-aggregate per second before writing.

Scoring falls back to the degraded policy (local rules and cached block lists), payments continue with tighter limits for risky segments, and on-call is paged. Features rebuild from Flink checkpoints and Kafka replay when it recovers.

Block rate jumps from 0.3% to 4% within minutes. Automated guards compare live decision rates with the expected range and roll back to the previous model version or switch the broken feature to a default value. All models are versioned, and rollback is a config change.

Velocity features go stale, and burst attacks may slip through. Alert on consumer lag in seconds; while lag is high, the scoring service tightens thresholds for new devices and high amounts until features are fresh again.