Design a news feed (Facebook)

Fan-out on write (push) copies each new post ID into every follower's inbox. Reads become a single fast list read, which matters because reads outnumber writes ~20:1. But a page with 100M followers would cause 100M writes per post and delay delivery for minutes. Fan-out on read (pull) stores posts only with the author and merges at read time: no write amplification, but a user following 500 sources needs 500 lookups per feed load. The hybrid used in practice: push for normal authors (say < 10k followers), pull for celebrities and big pages, and merge both at read time. A user typically follows only a handful of celebrities, so the read-time merge touches maybe 10-50 outboxes, which are hot in cache anyway. Two more optimisations: skip fan-out to inactive users (no login in 30 days) and rebuild their feed by pull when they return, and do fan-out in priority order (online users first). Numbers to say: ~1.2M inbox inserts/s is fine for a Redis cluster of a few hundred nodes; 100M inserts for one post is not.

Chronological feeds are simple but users see low-value posts. Ranked feeds score each candidate by predicted engagement: P(like), P(comment), P(share), expected dwell time, P(hide), combined with weights into one score. The work must fit inside ~100 ms. So ranking is staged: candidate generation (~500 IDs from inbox + outboxes + recommendations), a cheap first-pass model (logistic regression or small tree model) to cut to ~100, then a heavier neural model on the top ~100, then business rules: diversity, freshness boost, demotion of clickbait, integrity filters. Features come from a feature store: user features (precomputed daily and updated in streaming), author affinity (how often this user interacts with this author), and post features (age, early engagement velocity from Flink counters). Scoring is batched on GPU or optimised CPU inference servers. If ranking times out, the feed falls back to the inbox order with a simple heuristic, so the user still gets a feed. Logging impressions and outcomes back to Kafka closes the training loop.

Store only IDs in the inbox; never copy post bodies into millions of inboxes, because edits and deletes would have to touch every copy. A Redis sorted set per user with score = time (or Snowflake ID) and a cap of ~500-1,000 entries keeps memory at ~4-8 KB per user. For users beyond the cap, deeper history is pulled from authors' posts tables. Pagination is tricky with ranking: if you re-rank on every page, new posts shift positions and users see duplicates or miss posts. Solution: on the first request, rank a larger batch (e.g. 200), store that ordered list in a session cache keyed by a feed session ID, and serve pages from it using an opaque cursor (session_id + offset). New posts that arrive while scrolling are not inserted mid-list; they show up as a 'new posts' pill and are added at the top on refresh. Seen-post tracking (a per-user Bloom filter for the last 48 h) stops the same post from reappearing across sessions.

Fan-out happens before the reader asks, but permissions can change: a user unfriends, blocks, or the author changes the audience or deletes the post. So the inbox is only a candidate list, and every candidate is checked at read time: is the post deleted, is the viewer in the audience, has either side blocked the other. These checks use cached graph edges and post metadata in one batched multi-get, so they add a few ms. On delete, we do not chase every inbox; we mark the post deleted and the read-time filter drops it, and the inbox entry ages out. This also makes the system safe against fan-out bugs: a wrong inbox entry can never leak a private post, because authorisation is enforced on the read path, not on the write path.

Push: fast reads, heavy writes, wasted work for inactive users, celebrity problem. Pull: cheap writes, slow and expensive reads. Hybrid gives fast reads for nearly everyone with bounded write cost. Choose hybrid with a follower threshold, and say the threshold is tuned by measuring.

Chronological is transparent, cheap and easy to paginate. Ranked improves engagement but needs ML infrastructure, makes pagination harder, and raises fairness questions. Many apps offer both (ranked default, 'Most recent' option).

Redis gives sub-ms reads but RAM is expensive at TB scale. Cassandra is cheaper per GB and durable but has higher tail latency. Common answer: Redis for active users with a cap, rebuild from source for the rest; or a tiered store.

A film star posts during a big release and 50M followers open the app. Pull model reads the same outbox millions of times: replicate hot outboxes across cache nodes and use local in-process caches in Feed Service for the top few thousand authors.

On New Year's Eve post volume spikes and fan-out workers fall behind. Scale workers on Kafka lag, prioritise online recipients, and allow inactive recipients to be skipped entirely.

Scoring 300 candidates for 350k requests/s is expensive. Cache ranked sessions, use a cheap first-stage model, and reduce candidates for users who scroll little.

Feed Service has a strict timeout (~80 ms) for ranking and falls back to a heuristic order (recency × affinity from cache). Users get a slightly worse feed, not an error.

Inboxes are a cache. Affected users get their feed rebuilt by pull: read the followed-authors list and merge recent posts. Slower for a few minutes, then inbox is repopulated.

Kafka redelivers the event from the last committed offset. ZADD of the same post_id is idempotent, so replays do not duplicate entries.