Design Twitter / X

The home timeline is the heart of the design. With reads ~100x writes, we precompute timelines: when a user tweets, push the tweet ID into each follower's Redis list. This is the approach Twitter described publicly: timelines in Redis, about 800 entries each, replicated 3 times. But an account with 100M followers would need 100M Redis writes per tweet, which takes minutes and wastes work on inactive followers. So we use a threshold: accounts above ~100k followers are not fanned out. Their tweets are pulled at read time from their user timeline and merged in. Because a user follows few such accounts, the merge is a handful of extra cache reads. Further tuning: only fan out to users active in the last ~30 days, rebuild a returning user's timeline by pull, and process fan-out for followers who are online first. Retweets are fanned out like tweets but with a 'retweet of' reference so the original is hydrated once. Fan-out is idempotent (dedupe by tweet ID) so Kafka replays are safe.

Users expect a tweet to be searchable seconds after posting, so a nightly batch index is not enough. Indexers consume TweetCreated from Kafka, tokenise text (with language detection, because Indian Twitter mixes Hindi, Hinglish and English), and append to an in-memory index segment. Twitter's Earlybird design used an index optimised for appends where newest documents are searched first, because most queries want recent tweets. The index is partitioned by tweet ID hash across many shards, and each shard is replicated for read throughput. A query fans out to all partitions of the recent tier (scatter-gather), each returns its top results, and a broker merges them. Time tiering keeps the hot tier small: the last few days in RAM/SSD, older tweets in an archive tier queried only if needed. Ranking combines text relevance, recency, author reputation and engagement. Deletes are handled by a deleted-ID set checked at query time, then merged out during segment compaction.

Trending is not 'most frequent', otherwise 'good morning' trends every day. It is 'growing much faster than usual'. A stream processor (Flink) reads tweets, extracts hashtags and n-grams, and counts them per location in 1-minute buckets. For each term it keeps a short window count (last 15 minutes) and a baseline (same hour over past days, or an exponentially decayed average). The trend score is roughly the ratio of current rate to expected rate, with a minimum volume and a boost for many distinct users (so one bot farm cannot trend a tag). Counting every term exactly is costly, so use a Count-Min Sketch plus a heap for top-K candidates, then exact counts only for candidates. Per location (India, Mumbai, Bengaluru) the top 30 are written to Redis every minute. Before publishing, a safety filter removes spam, abusive terms and coordinated manipulation; this step is a real requirement, not an afterthought.

Tweet IDs are 64-bit Snowflake IDs: 41 bits of milliseconds since a custom epoch, 10 bits of worker ID and 12 bits of sequence, so each worker can make 4,096 IDs per ms without coordination. Because IDs are roughly time-ordered, every list (home timeline, user timeline, search results) can be sorted and paged by ID. Pagination uses max_id/since_id cursors: 'give me tweets with ID < X'. This is stable when new tweets arrive at the top, unlike offset pagination, which would show duplicates. Clock skew between workers can produce slightly out-of-order IDs; this is acceptable for a timeline. Workers must never move their clock backwards: if NTP steps time back, the generator waits or refuses until time catches up.

Push makes reads cheap at the cost of write amplification and storage; pull makes writes cheap and reads expensive. Twitter's read:write ratio favours push with a pull exception for big accounts. State the threshold and that it is tuned from metrics.

Chronological ('Following') is predictable and easy to page. Ranked ('For you') adds out-of-network recommendations and engagement models, at the cost of a candidate-generation and ranking layer. Offer both; the storage layer above serves both.

Like and retweet counts and trend counts do not need to be exact in real time. Approximate, batched counters (or sketches) cut write load massively. Exact counts can be reconciled in batch.

During a World Cup final, tweets/s can jump 10x within seconds. Kafka absorbs the burst; fan-out lag grows briefly. Autoscale fan-out workers on consumer lag, and pre-scale before known events (IPL final, elections).

A viral tweet is read millions of times per minute. Cache it in Memcached with replicas and a small in-process cache in Timeline Service so one cache shard does not melt.

Every query hits every partition of the recent tier. As partitions grow, add replicas for throughput and keep the hot tier small through time tiering.

Timelines are a cache with replicas; if all copies are lost, the Timeline Service rebuilds by pull (merge recent tweets from followed users) and repopulates. Slower first load, no data loss.

If workers stall, tweets still appear in user timelines and search; home timelines are stale. Kafka keeps the events; workers catch up. Celebrity pull path is unaffected.

Queries return partial results from healthy partitions with a flag rather than failing. Replicas take over; a rebuilt partition replays from Kafka.