Design a recommendation system

You cannot score 50M items with a heavy model for each request within 150 ms. So the system is a funnel where each stage handles fewer items with more compute per item. Retrieval (candidate generation) must be cheap and high-recall: a two-tower model learns a user embedding and an item embedding whose dot product predicts interaction; item embeddings are precomputed and put into an ANN index (HNSW or IVF-PQ), and at request time you compute the user embedding and fetch the nearest few hundred items in ~5-10 ms. Several retrieval sources are used together because each covers a gap: co-occurrence for 'bought together', trending for freshness, recent views for intent. Ranking is precision-focused: a model with hundreds of features, including cross features like 'user's affinity to this brand' that a two-tower model cannot express. Re-ranking handles list-level goals that a pointwise model ignores: diversity, de-duplication, fairness to sellers, sponsored placements. Say the numbers: 50M -> 500 -> 100 -> 20.

The most common silent bug in recommender systems is that the model sees different feature values in training than in serving. Example: training computes 'clicks in last 7 days' from the warehouse with end-of-day data, while serving reads a streaming counter that includes the last minute; or training accidentally uses data from after the impression (leakage), so offline AUC looks great and online results are flat. A feature store addresses this: features are defined once, computed by the same code for batch and stream, written to an offline store with timestamps and to an online store for serving. Training data is built with point-in-time joins: for an impression at 10:05, use feature values as of 10:05. Another robust practice is to log the exact features used at serving time with the request_id, and train on those logged features. Monitor feature distributions online vs training and alert on drift or missing values.

New users have no history. Use what you do know: signup context (city, language, device, acquisition campaign), onboarding choices, and the first few clicks of the session, which streaming features capture quickly. Until then, show popular items by city and segment. New items have no interactions, so collaborative signals give them nothing. Use content-based embeddings (from title, category, image, price) so the item tower can place a new item near similar ones from day one, and reserve exploration slots: for example one of the top 10 positions goes to new or uncertain items selected by a bandit (Thompson sampling or epsilon-greedy). Without exploration you create a feedback loop: the model only shows items it already likes, collects data only on them and popular items get more popular. Log the propensity (probability an item was shown) so off-policy evaluation and debiasing are possible later.

Offline metrics are necessary but not sufficient. For retrieval, measure recall@K: of the items the user actually engaged with next, how many were in the candidate set. For ranking, AUC, log loss and NDCG on a time-split holdout (train on days 1-28, test on day 29, never random splits that leak the future). But offline gains often do not translate online because logs only contain items the old system showed. The real decision comes from A/B tests on the business metric: orders or revenue per user for commerce, watch time or retention for video. Watch guardrails: latency, return rate, diversity, complaint rate, seller concentration. Run tests long enough to cover weekday/weekend cycles and beware novelty effects. Optimising only CTR leads to clickbait; combine predicted click, purchase and satisfaction (e.g. not returned) into the final score with tuned weights.

Precomputing top-N per user nightly is cheap to serve but ignores the session and wastes work on inactive users. Online computation reacts to clicks in seconds but needs low-latency features and models. Common hybrid: precompute candidates and embeddings, rank online.

GBDTs (LightGBM, XGBoost) are fast on CPU, strong on tabular features and easy to debug: a good v1. Deep models (DLRM, DCN, transformers over sequences) win at large scale with sparse IDs and sequence features but need GPUs and more MLOps.

Real-time session features make recs responsive, but too much weight on the last click makes the feed jumpy. Blend long-term and session embeddings and cap session influence.

Optimising one signal (click) is simple but leads to clickbait. A weighted combination of p(click), p(purchase) and p(satisfied) aligns with business goals; the weights become product decisions tested in A/B.

Scoring 500 items x 50k QPS is heavy. Cut candidates per stage, cache user features per session, batch requests on GPU servers, and use a distilled smaller model at peak. Cache the full result for 30-60 s per user per surface.

Popular items' features are read by almost every request. Keep a local in-process cache of item features with a short TTL; only user features need a remote read per request.

Impressions grow with traffic. Downsample negatives (impressions without clicks) and correct with weights; keep a rolling window.

Orchestrator has per-stage deadlines. If ranking misses its 40 ms deadline, return candidates ordered by retrieval score; if retrieval fails, serve a precomputed popular-in-city list from cache.

Canary on a small traffic slice with automatic guardrail checks (CTR collapse, latency, empty results). Model registry keeps the previous version for instant rollback.

Features carry an updated_at; the ranker treats stale session features as missing (the model is trained with missing values), so quality degrades gently instead of using wrong data.