Design video conferencing (Zoom / Google Meet)

There are three ways to connect participants. In a mesh, every client sends its stream directly to every other client. With n people, each uploads n-1 copies; at 1.5 Mbps that is 13.5 Mbps upload for 10 people, which phones and home connections cannot do. Mesh is fine only for 2-3 people (and 1:1 calls often use peer-to-peer to save server cost). An MCU (Multipoint Control Unit) decodes all incoming streams, mixes them into one composite video, re-encodes and sends one stream to each participant. Clients are simple, but the server does heavy transcoding, which costs a lot of CPU, adds latency (decode plus encode, often 100+ ms) and fixes the layout. An SFU (Selective Forwarding Unit) receives each participant's stream once and forwards packets to subscribers without decoding. It is cheap per stream, adds only a few ms, and lets each client choose its own layout. Its weakness is download bandwidth on the client grows with visible tiles, which is solved with simulcast and limiting visible tiles. Modern systems (Zoom, Meet, Teams) are SFU-based, sometimes with MCU-style mixing for PSTN dial-in or very constrained devices.

Participants have very different networks: office fibre, home Wi-Fi, a 4G phone on a train. Real-time video cannot buffer like YouTube, so it must adapt within a second. With simulcast, each sender encodes the same video at 2-3 resolutions and bitrates and uploads all of them; the SFU picks per receiver which layer to forward. With SVC (scalable video coding, available in VP9 and AV1), one stream contains layered data, and the SFU drops upper layers for weaker receivers. Bandwidth estimation runs on both ends using RTCP feedback and transport-wide congestion control: the receiver reports packet arrival times, the sender's estimator (like Google Congestion Control) detects rising delay before loss and lowers its bitrate. Under heavy loss, priorities apply: audio is protected first (Opus with FEC, small packets, retransmission only if RTT allows), then screen share (needs sharp text at low frame rate), then speaker video, then thumbnails. NACK-based retransmission fixes single lost packets; keyframe requests (PLI) recover from bigger losses. The user sees a lower resolution rather than a frozen call, which is the right trade-off.

Latency is mostly physics, so media servers must be near users. Deploy SFUs in many regions (Mumbai, Chennai, Singapore, Frankfurt, Virginia) and edge PoPs. When a meeting starts, the allocator picks an SFU near the first participant. If everyone is in India, all connect to Mumbai. If a participant joins from the US, connecting them to Mumbai adds ~200+ ms round trip over the public internet. Cascading solves this: the US participant connects to a US SFU, and the two SFUs connect to each other over the provider's private backbone, forwarding only the streams that are actually subscribed on the other side. This cuts both latency and cross-region bandwidth. Large meetings also need cascading because one SFU has limits (around 500-1,000 participants or ~10 Gbps); a 300-person meeting can be split across several SFUs in the same region. The allocator uses live load from the SFU registry and avoids moving a running meeting; if an SFU is overloaded, new joiners go to a cascaded sibling.

Most devices sit behind NAT, and many corporate networks block UDP. ICE tries candidates in order: direct host addresses, server-reflexive addresses learned from STUN, and finally relayed addresses through TURN. Because clients connect to public SFUs, direct UDP to the SFU usually works; TURN (often over TLS on port 443 so it looks like HTTPS) handles the 10-20% of users behind strict firewalls, at the cost of extra latency and relay bandwidth, so TURN servers must be co-located with SFUs. Media is encrypted hop by hop with DTLS-SRTP, so the SFU can see media in clear. For true end-to-end encryption, clients encrypt frames with a shared key using insertable streams (SFrame) before packetising; the SFU still forwards because it only needs RTP headers, but server-side features like recording, transcription and PSTN mixing then stop working, which is why E2EE is usually an opt-in mode. Join tokens are short-lived and bound to the meeting, waiting rooms and passcodes stop 'zoombombing', and host controls are enforced by the signalling server, not trusted from clients.

TCP retransmits everything in order, so one lost packet stalls all later packets (head-of-line blocking). For live media, late data is useless, so RTP over UDP with selective recovery is used. TCP/TLS via TURN is only a fallback.

Simulcast works with H.264 and VP8 everywhere but costs extra upload bandwidth and encode CPU. SVC is more efficient but needs VP9/AV1 support on all clients and in hardware. Many systems use simulcast with VP8/H.264 and add SVC where supported.

P2P saves server bandwidth for two-party calls and can lower latency. But switching to SFU when a third person joins causes a renegotiation glitch, and recording needs a server. Many products use P2P for 1:1 and switch on the third join.

For 10,000 passive viewers, an SFU fan-out is expensive. Converting the meeting into a low-latency live stream (LL-HLS, 2-5 s delay) through a CDN is far cheaper, at the cost of interaction latency.

Meetings start on the hour. Pre-warm SFU capacity by time of day and region, spread allocation, and keep join fast by not waiting on non-critical calls (recording setup, chat history).

Decoding 25 video tiles burns phones. Cap visible tiles on mobile, subscribe to low layers for thumbnails, and pause video for off-screen participants.

Cascaded links carry only subscribed streams and lower layers where possible; most meetings are kept in one region.

Clients detect ICE failure within a few seconds and signalling re-allocates the room to a healthy SFU; clients do an ICE restart and republish. The call freezes for 2-5 s but continues. Room state lives in signalling and Redis, not in the SFU.

Existing media keeps flowing because it does not depend on signalling. Clients reconnect their WebSocket to another signalling node, which loads room state from Redis.

Their outgoing video drops to the lowest layer or turns off, audio uses FEC, and the UI shows a 'poor connection' indicator. Other participants are unaffected because the SFU isolates each receiver.