Design Netflix

With a known catalogue and predictable demand, Netflix does not wait for cache misses. Its Open Connect appliances (storage-heavy servers given free to ISPs or placed at internet exchanges) are filled every night during off-peak hours with the files each location is predicted to need. Popularity is very skewed, so a box with a few hundred TB holds the content behind most of that ISP's viewing. Fill uses a tiered approach: appliances can fill from peers, reducing load on the cloud origin. At play time, the steering service in the cloud chooses appliances for each client: the appliance must hold the needed files, be healthy, have spare capacity and be 'close' in network terms (appliances report the BGP routes the ISP gives them, so steering knows which client prefixes each appliance serves best). The client gets several ranked URLs and fails over between them on its own. The result: at prime time, around 95%+ of bytes are served from inside or next to the viewer's ISP, latency is low, and the ISP saves transit cost, which is why ISPs agree to host the boxes.

A fixed bitrate ladder (say 1080p always at 5.8 Mbps) wastes bits on a simple cartoon and starves a grainy action film. Netflix moved to per-title encoding, choosing the ladder for each title by measuring quality (VMAF, a perceptual metric) at many resolution/bitrate points and picking the convex hull. It later went further to per-shot (dynamic optimizer) encoding, where each shot gets its own encoding parameters. The result is the same visual quality at noticeably lower bitrate, which matters for mobile users in India on data plans and for total egress. The pipeline splits the studio mezzanine file into chunks, encodes chunks in parallel on a large batch compute fleet, validates every output (automated checks for artefacts, sync, missing frames), and packages encrypted segments. Because each title is encoded once and watched millions of times, it pays to spend heavy compute: AV1 and HEVC encodes for capable devices, H.264 for compatibility, and special low-bitrate mobile encodes.

Everything before the first video byte runs in the cloud and must be fast and highly available, because a failure means no playback at all. The manifest call evaluates entitlement (plan, region availability windows, maturity rules) and device capability, so the response lists only streams the device can decode and the member may watch (4K only on the premium plan). The licence call issues DRM keys: Widevine for Android and Chrome, FairPlay for Apple, PlayReady for many TVs and Windows. Licences are bound to the device and expire, and offline downloads carry licences with longer expiry and a download count limit. Concurrent stream limits are enforced with heartbeats and a TTL entry per active session: a fifth device on a four-screen plan is refused, and a crashed device's slot frees itself after the TTL. These services are stateless, replicated in several AWS regions, and backed by replicated caches so a regional failure does not stop playback.

The control plane is hundreds of microservices, so something is always failing. Patterns that keep the product up: every service call has a timeout, a circuit breaker and a fallback (if the personalised row service is down, show popular rows; if the bookmark read fails, start from the beginning rather than refusing to play). Critical paths (playback start, licence) are kept as short and dependency-light as possible, while non-critical calls are made optional. Requests carry priority so that under overload the gateway sheds low-priority traffic (prefetching artwork, logging) before playback. The whole control plane runs active-active in several regions with Cassandra and EVCache replicated across them; if a region degrades, traffic is shifted by DNS and gateway rules within minutes. Netflix popularised chaos engineering (Chaos Monkey and region evacuation drills) precisely to prove that these fallbacks work before a real failure tests them.

Pull-through caching (fetch on first miss) is simpler and fits a huge long-tail catalogue like YouTube. Netflix's small, predictable catalogue makes proactive fill better: no prime-time miss storms and fill traffic moved to off-peak hours when bandwidth is cheap. The cost is a prediction system and a little wasted storage.

At hundreds of Tbps, owning the appliances is cheaper and gives full control over steering and protocols. It requires partnerships with thousands of ISPs and a hardware operations team, which only makes sense at very large scale.

Precomputing rows offline makes the home API fast and resilient but can be stale. A hybrid applies cheap real-time adjustments (continue watching, remove just-watched titles) on top of the precomputed page.

Per-shot and AV1 encoding cost much more compute per title, but each title is streamed millions of times, so bandwidth savings dominate. For a long-tail UGC platform the maths is different.

A global release at a fixed time sends millions of plays in the first hour. Pre-fill the episodes everywhere days before, pre-scale playback and licence services, and stagger non-essential calls such as artwork prefetch.

If an ISP's appliances run out of capacity, steering spills clients to the nearest exchange-point cluster. Health and load are reported continuously so steering reacts in seconds.

Heartbeats every minute from 45M streams are ~750k writes/s. Cassandra partitioned by profile id spreads them; keep writes small and use LWW columns instead of read-modify-write.

The player already holds alternative URLs; it retries the next segment range from another appliance. The viewer may see a short quality dip, not a stop. Steering stops sending clients to the failed box within seconds.

Traffic for the control plane is evacuated to other regions (all regions run active-active with replicated data). Streams already playing continue from appliances; new plays route to healthy regions.

The home service falls back to cached country-level popular rows. The page looks less personal but works.

Licences are short calls with strict timeouts and retries across regions. Recently issued licences remain valid for their lifetime, so ongoing playback is not affected.