A practical guide to live streaming protocols, latency and scaling

Live video looks simple until you build it. Then you discover that "low latency" means five different things, that your CDN and your latency target are fighting each other, and that the box which handled ten viewers falls over at ten thousand for reasons nobody warned you about. This is the guide I wish existed when I started. No vendor talk, just how the pieces fit. 1. Ingest and delivery are separate decisions The single most common mistake is treating "streaming protocol" as one choice. It is two. Ingest is getting video from a camera, encoder or browser into your server. Delivery is getting it from your server to viewers. They have different constraints and you almost never use the same protocol for both. A typical stack ingests over RTMP or SRT and delivers over HLS. Another ingests WebRTC and delivers WebRTC. Mixing is normal and expected. Once you separate them, most of the confusion disappears. 2. The ingest protocols RTMP is old, TCP-based, and still everywhere. Every encoder speaks it, OBS defaults to it, and it just works. Latency is typically 2 to 5 seconds. Classic RTMP is limited to H.264 and AAC, though the Enhanced RTMP spec has added HEVC and AV1. Being TCP, it degrades badly on lossy networks: packet loss becomes head-of-line blocking, and your stream stalls instead of gracefully dropping quality. SRT is the answer to that. UDP-based with its own retransmission layer (ARQ), a configurable latency buffer, and built-in AES encryption. It is designed for pushing broadcast-quality video across the public internet, which is exactly where RTMP struggles. If your source is on a flaky connection, a 4G link, or a different continent, SRT is usually the right call. # Publishing over SRT with ffmpeg ffmpeg -re -i input.mp4 -c copy -f mpegts \ "srt://your-server:4200?streamid=live/stream1" RTSP is what IP cameras speak. If you are pulling from surveillance hardware, you are pulling RTSP whether you like it or not. WHIP (WebRTC-HTTP Ingestion Protocol) is the newer option. It standardises WebRTC signalling over plain HTTP, which historically was the messy part of WebRTC. If you want browser-based publishing with sub-second latency and no plugin, this is the modern path. 3. The delivery protocols HLS and DASH chop your stream into segments and serve them over plain HTTP. That last part is why they dominate: HTTP means any CDN can cache them, which means they scale to millions of viewers cheaply. The cost is latency, typically 10 to 30 seconds. LL-HLS and LL-DASH cut that down using partial segments, blocking playlist reloads and preload hints. Expect 2 to 6 seconds while keeping CDN compatibility. For most "live" experiences this is the sweet spot. WebRTC, and WHEP for standardised playback, gets you under half a second. It is the only real option for genuine interaction: auctions, betting, two-way audio, remote control of anything. The catch is that it is not natively cacheable by traditional CDNs, so scaling works differently (more on that below). MoQ (Media over QUIC) is the interesting one to watch. It aims to give you low latency and CDN-style scaling by building on QUIC. Still an IETF work in progress, not something to bet a production launch on yet, but worth following. 4. Where latency actually comes from This is the section that saves people the most time, because most latency is not where they think. Your end-to-end latency is a sum: encoder buffer + network transit to server + server processing (transcoding, packaging) + segment duration + playlist/manifest window + CDN propagation + player buffer = what the viewer sees Two of those dominate for HTTP-based delivery. Segment duration. Apple's classic recommendation was 6-second segments. Players typically buffer three segments before starting playback. That is 18 seconds of latency before you have transcoded a single frame or crossed a single network hop. Player buffer. This is the silent killer. You can tune everything server-side and then discover the player is holding 15 seconds because that is its default. Always check what the client is doing before you blame the pipeline. The practical implication: if someone tells you their HLS latency is 30 seconds and asks you to fix the server, the server is probably not the problem. Shorter segments, a smaller playlist window, and a tuned player will get you most of the way. And if you need to go below roughly 2 seconds, no amount of tuning helps. Segmented delivery has a floor. You have to change protocol. 5. Choosing: a decision table Your requirement Protocol Realistic latency Interactive, two-way, betting, auctions WebRTC / WHEP under 0.5s Live sport, "feels live", large audience LL-HLS 2 to 6s VOD-style live, maximum reach, lowest cost HLS / DASH 10 to 30s Contribution over unreliable networks SRT (ingest) configurable Anything an existing encoder must publish to RTMP (ingest) 2 to 5s Pick the highest latency you can actually tolerate. Every step down the table costs you money, CDN compatibility, or both. 6. Scaling: two completely different problems Live streaming workloads split into two shapes, and they scale in opposite directions. Few publishers, many viewers. A football match to 100,000 people. This is an egress problem. Your ingest server is barely working. You solve it with an origin-edge topology and a CDN in front, and the CDN does the heavy lifting. Many publishers, few viewers. Eighty security cameras with three people watching. This is an ingest and transcode problem. Your origin is saturated and your edges are idle. A CDN buys you almost nothing here. People routinely size for the wrong one. Work out which shape you are before you provision anything. Transcoding is where your CPU goes Passing a stream through untouched is cheap. Transcoding is not, and adaptive bitrate multiplies it. An ABR ladder of 1080p, 720p, 480p and 360p means encoding four renditions of every incoming stream. Ten input streams becomes forty encodes. This is usually the single largest cost in the system, and it is why hardware encoding (NVENC, QuickSync, or similar) matters once you pass a modest number of streams. Rule of thumb worth internalising: count your renditions, not your streams. WebRTC scales differently Because WebRTC is not cacheable by conventional CDNs, you scale it with cascading SFUs. Origin ingests, edges subscribe from origin, viewers connect to the nearest edge. It works and it scales to large numbers, but you are running and paying for that fleet yourself rather than renting a CDN's. This is the real cost of sub-second latency, and it is worth pricing before you commit to it. 7. Server location beats everything Worth stating plainly because it comes up constantly: your latency is determined by where your servers are relative to your viewers. Not where your software vendor is based, not where you happen to be sitting. If your audience is in Brazil, deploy in Brazil. If it is spread across three continents, you need edges on three continents. This is a topology decision and no protocol choice will rescue a bad one. The one question to answer first Before comparing anything, decide this: What is the single number that makes this project succeed or fail? Sub-second latency. Cost per streaming hour. Concurrent viewers at peak. Rights compliance. Time to launch. Almost every decision above falls out of that one number, and almost nobody picks it before they start evaluating tools. Pick it first and the rest of the architecture more or less designs itself. I work on the solutions side at Ant Media and spend most of my time on exactly these conversations. We are at IBC2026 in Amsterdam, 11 to 14 September, if you want to talk architecture in person. Otherwise the comments work fine, happy to go deeper on any of this.