A look at how Zoom works. Zoom handles millions of concurrent video meetings per day with sub-second latency, and the architectural decision that makes it possible is the choice of SFU (Selective Forwarding Unit) over the more familiar MCU or peer-to-peer mesh. That one choice is why Zoom scales to 1,000-person meetings while competitors stall at 30.
How Zoom Works: The problem at scale
Real-time video has three candidate architectures, and two of them fail at scale. Peer-to-peer mesh (every participant connects to every other) works for 3-4 people but burns O(N²) bandwidth per participant: at 10 people you’d need 9 upstream video streams per device. MCU (Multipoint Control Unit) centralizes: one server decodes everyone’s video, composites a single layout, re-encodes, sends to everyone. Works for small meetings but the central compositor is a CPU bottleneck, and you can’t let each participant pick their own layout. SFU is the middle path: the server forwards packets without decoding them, each participant sends one upstream and receives N-1 downstreams. The server is a router, not a decoder: far cheaper per participant.
The core architecture
When you join a meeting, your client opens a UDP connection to the nearest Zoom Edge POP (multi-region, low-latency). You send one encoded video stream and one encoded audio stream upstream. The Video Router (SFU) receives them and forwards them to every other participant’s edge POP, which forwards to their clients. You receive N-1 video streams (one per other participant) and one mixed audio stream. The Audio Mixer (MCU) decodes all audio streams, sums them into a single mix (with echo cancellation and noise suppression), re-encodes, and sends to everyone. This is necessary because mixing audio requires decoding, and you don’t want each client doing N-1 decodes. The Active Speaker Detect service watches audio energy per stream and broadcasts “who’s currently speaking” so clients can feature the right tile.
Why SFU beats MCU at scale
The SFU’s CPU cost per participant is roughly constant (just packet forwarding, no decode/encode), while MCU’s cost grows linearly with participant count (decode everyone, composite, re-encode). At 100 participants, the MCU server is doing 100 decodes + 1 composite + 100 re-encodes per frame; the SFU is doing 100 packet forwards per frame. That’s why Zoom can support 1,000-person webinars while older conferencing systems capped at 25. The bandwidth trade-off is also favorable: each participant receives N-1 streams but only at the resolution they need (Zoom sends lower-resolution versions of non-speaker videos), keeping total downstream manageable.
Active-speaker detection drives the UX
The active-speaker service is a small ML model on the audio energy per stream, with a smoothing window (typically 200-500ms) to avoid flicker. When the active speaker changes, every client gets a notification and the UI reflows: speaker view features the new speaker, gallery view highlights them. This is what makes Zoom’s UI feel responsive despite the underlying stream forwarding being relatively dumb: the intelligence is in the active-speaker signal, not the video transport.
The one non-obvious insight
Zoom uses UDP, not TCP, for both audio and video. This sounds scary (UDP doesn’t guarantee delivery) but is essential for real-time: a lost video frame is better dropped than retransmitted (a retransmitted frame arrives too late to be useful and just adds latency). TCP’s reliability guarantees would actively hurt real-time video. Zoom’s protocol (a proprietary variant built on WebRTC primitives) implements its own congestion control and packet-loss concealment: dropping frames gracefully when bandwidth drops, prioritizing audio over video when networks are constrained. This is why a Zoom call stays intelligible on a bad connection where a TCP-based video call would freeze.
Zoom is the canonical SFU video-conferencing architecture. By forwarding packets instead of decoding them, the server stays cheap per participant and scales to 1,000-person meetings. UDP transport plus active-speaker detection is what makes it feel responsive on any network.
Continue learning: