Design ML-personalised notifications

If you rank by predicted open rate alone, the model learns to send more clickbait and more notifications, which raises opens this week and opt-outs next month. The decision must weigh value against harm. Train several heads (or models) on logged outcomes: p_open, p_convert within 24 hours, and p_negative (disable notifications, mute category, uninstall within 7 days). Score each candidate as expected value minus expected cost: p_convert × value_per_conversion − λ × p_negative × value of keeping the channel − channel cost (WhatsApp and SMS cost money; push does not). λ is tuned with experiments, not guessed: run arms with different λ and pick the one that maximises long-term orders, not 24-hour clicks. Then apply a per-user budget: a light user who rarely opens should get fewer notifications, and the budget itself can be learned (users with high p_negative get a lower cap). Choosing items across many campaigns under a budget is a small knapsack problem per user; greedy by utility with diversity rules (no two food-discount pushes in a row) is good enough. Hard caps and quiet hours sit outside the model as rules, because they are product promises, not predictions.

Attribution says 'the user opened the push and ordered, so the push caused ₹400 of revenue'. Often the user was going to order dinner anyway; the push only claimed credit. To know whether notifications add value, keep a randomised holdout: e.g. 5% of users get no promotional notifications for a quarter (transactional ones still flow). Comparing orders per user between holdout and treated users gives the true incremental lift, with a confidence interval. Per campaign, use a smaller campaign-level holdout. The model should also optimise for incremental conversions, which is an uplift-modelling problem: predict p_convert if sent minus p_convert if not sent, trained using the randomised exploration data. Exploration matters for another reason: the model only learns from what it chose to send, so without randomness it never learns about candidates it always ranks low (feedback loop). Send ~5% of decisions using an exploration policy (epsilon-greedy or Thompson sampling across variants) and log the propensity of every decision; that enables off-policy evaluation (inverse propensity scoring) of a new model on old logs before an A/B test. Strong candidates bring up holdouts without being asked.

Notification models depend on fresh behaviour: a user who ordered 10 minutes ago must not get 'hungry?'. Features come in three speeds. Batch features (30-day order counts, category affinity, open-hour histogram) are computed nightly in Spark and loaded into the online store. Streaming features (last order time, last app open, promos sent today) are updated by a Flink or Kafka Streams job within seconds. Request-time features (current cart value, time of day) come with the trigger. The classic bug is training-serving skew: the training pipeline computes 'orders in last 30 days' slightly differently from the online path, or uses values as of today for an event that happened last week (leakage), and the model looks great offline and fails online. A feature store (Feast-style, or in-house) solves this by defining each feature once, materialising it to both offline (Iceberg) and online (Redis) stores, and generating training data with point-in-time-correct joins: for each logged decision, features as they were at decision time. Log the exact feature vector used at serving time as well; it is the cheapest way to debug skew.

Decisions made at 6 am are executed hours later, and the world changes. The scheduler re-checks each decision just before sending: is the user still opted in, did a higher-priority trigger already use the budget (atomic check-and-increment on the fatigue counter in Redis), did the user already order, is the campaign still active, is the restaurant open, is it inside quiet hours for the user's time zone? This prevents the embarrassing 'complete your order' after an order. Idempotency: the decision key (user_id, candidate_key) is unique in Cassandra, and the notification service dedupes on decision_id, so a scheduler crash and retry cannot send twice. Pacing: send-time optimisation pulls many sends into the same popular minutes (7:30 pm), and a campaign launch can schedule millions at once. The scheduler spreads sends within a few-minute window and drains per-minute buckets at a controlled rate so FCM/APNs quotas, WhatsApp/SMS vendor throughput and our own backend (the app's API gets hit when millions open at once) are not overwhelmed. For SMS in India, only DLT-registered templates can be sent, so LLM-generated copy is only usable for SMS after template registration.

Scoring everything online gives the freshest decisions but means 500M model calls through a low-latency path. Batch scoring for scheduled campaigns is far cheaper and enough because decisions are re-validated at send time; real-time scoring is reserved for triggers where minutes matter.

Per-user live generation could personalise wording, but it is costly at 70M sends/day, slow, hard to review for brand and legal compliance, and can hallucinate discounts. Generate a few variants per campaign, segment and language offline, have them reviewed, use placeholders for facts (₹{discount}, {restaurant}), and let a bandit pick the variant per user.

A single multi-task model with channel as a feature shares learning and is easier to run; per-channel models capture very different behaviour (WhatsApp vs push) but fragment data. Start with one multi-task model and split only when evaluation shows a clear gain.

Sending 10M pushes in 5 minutes makes millions of users open the app at once, overloading menus and search. Pace sends, coordinate with backend capacity, and stagger by city and cohort.

Trigger bursts at dinner time hit Redis hard. Batch feature reads per user, colocate a user's features in one hash, and keep the trigger path's feature set small.

Transactional messages are unaffected (separate path). For promos, fall back to a simple rule-based policy (campaign priority + caps) or delay sending; never fall back to 'send everything', which spikes opt-outs.

Hard caps outside the model bound the damage; guardrail metrics (opt-out rate, sends per user) are monitored per model version with automatic rollback to the previous version.

Decisions in Cassandra have status scheduled; on restart the scheduler reprocesses due buckets. Unique decision ids and the delivery system's dedupe make the retry safe.