Design YouTube

Encoding a 2-hour 4K video serially into ten renditions would take many hours. Instead, the orchestrator splits the source into independent chunks at keyframe (GOP) boundaries, typically a few seconds to tens of seconds each, and encodes every (chunk, rendition) pair as a separate task across thousands of workers. A merge step concatenates the encoded chunks and a packager produces CMAF segments with DASH and HLS manifests. Model it as a DAG in a workflow engine (Temporal or an internal scheduler) so each task is retried independently, progress is checkpointed, and a worker crash costs one chunk, not the whole video. Prioritise: produce 360p first so the video is watchable within minutes, then higher resolutions. Spend compute where it pays off: every upload gets H.264 for compatibility and VP9 for mid-popularity, but AV1 encoding is expensive and is done only after a video crosses a view threshold, because the bandwidth saving (~30% vs VP9) only matters when many people watch. Workers should be spot/preemptible instances, since tasks are idempotent and short. Keep the original upload so you can re-encode the catalogue when a better codec arrives.

Video is cut into short segments (2-6 s) available at several bitrates. The player picks the next segment's bitrate from measured throughput and buffer health (buffer-based algorithms like BOLA work well on noisy mobile networks). Short segments react faster but add request overhead; 2-4 s is a common balance. Because segments are immutable files with stable URLs, they cache perfectly. Delivery is tiered: edge caches inside ISPs (Google Global Cache, Netflix Open Connect style), regional PoPs, a mid-tier origin shield, and finally origin storage. Head content (the top few percent of videos, which draw most views) is almost always served from the edge; the long tail is fetched on demand and may come from a regional tier with slightly higher start time. The playback service steers each client to a good CDN node by IP, ISP and node load, and signs URLs with a short expiry so links cannot be hot-linked forever. For India, publishing low rungs (144p at ~80-100 kbps, 240p) and starting at a conservative bitrate cuts startup failures on congested networks.

Video metadata is small (a few KB per video) but read very often. YouTube scaled MySQL by building Vitess, which shards tables transparently, routes queries by a keyspace id, and handles resharding and connection pooling. Shard videos by video_id so reads are spread evenly; channel pages use a secondary table keyed by channel_id. Put a cache (Memcached/Redis) in front: the read/write ratio is extreme (a video is written a few times and read millions of times), so a 95%+ hit rate is normal. Do not store fast-changing counters (views, likes) in the same row: they would cause write hot spots and cache invalidation storms. Keep them in a separate counters store and merge them in the API layer. Video ids are random (not sequential) so new uploads do not all land on one shard and ids cannot be enumerated.

Incrementing a database counter on each view is ~60k writes/s on average with extreme hot keys: a viral music video can get 50k views/s on one row. Instead, the player sends watch events to a collector, events go to Kafka partitioned by video_id, and a stream processor (Flink) validates and aggregates them. Validation matters because view counts drive money and ranking: a view counts only after a minimum watch time, repeated reloads by the same user or IP within a window are dropped, and known bot patterns are filtered. The aggregator flushes per-video sums every 10-30 seconds, turning millions of events into thousands of writes. For a super-hot video, pre-aggregate inside each partition and shard the counter (several sub-counters summed on read). The public count can be eventually consistent; YouTube itself has historically frozen or delayed counts while validating suspicious spikes. Exact numbers for creator payouts are computed later in batch from the warehouse.

Encoding every rendition and codec for every upload wastes compute: most videos get few views. Encode a baseline ladder for all and add expensive codecs (AV1, 4K HDR) only when popularity justifies it. The trade is slightly worse quality-per-bit for long-tail videos in exchange for large compute savings.

2 s segments adapt quickly and start faster but mean more requests and worse compression (more keyframes). 6-10 s segments are efficient but adapt slowly on mobile. 2-4 s is a good default for VOD.

At ~100 Tbps, placing your own caches inside ISPs is far cheaper and faster than paying per GB, and ISPs welcome it because it cuts their transit. A smaller platform should start with commercial CDNs and multi-CDN steering.

Exact real-time counts need transactional increments on hot rows. Approximate, slightly delayed counts from stream aggregation scale easily and allow fraud filtering first. Creator payouts use exact batch numbers.

A sudden spike causes many simultaneous edge misses for the same segment. Request coalescing at the edge and the origin shield (one upstream fetch, many waiters) plus pre-warming edges for scheduled premieres keep origin load flat.

Upload peaks (evenings, festivals) can outgrow the encoding fleet. Autoscale on queue depth with spot instances, prioritise low renditions and partners, and let high renditions catch up later.

Petabytes per day accumulate forever. Move old originals and rarely watched renditions to colder storage classes, delete renditions that can be regenerated, and use erasure coding instead of 3x replication.

The workflow engine notices the missed heartbeat and reschedules that chunk task on another worker. Outputs are written to deterministic keys, so a retried task simply overwrites; the video is delayed by seconds, not reprocessed.

The playback service and DNS steer clients to the next-best node; players retry a failed segment against an alternate host listed in the manifest. Viewers may see one quality drop, not an error.

Serve video pages from cache (slightly stale) while Vitess fails over to a replica. Playback of already-started videos is unaffected because segments come from the CDN.

Resumable uploads keep acknowledged byte ranges; the client queries the session and continues from the last offset. Incomplete sessions expire after a few days.