The Evolution of Low-Latency Video Streaming

by Dec 15, 2021CMMA Blog, https://www.brightcove.com/en/blog/tech-talk0 comments

In the past few years, the video streaming industry has seen immense interest in low-latency streaming protocols. The majority of interest is on delivering videos with a sub-five-second delay, making them comparable with delays in live broadcast TV systems. Attaining such low delay is critical for streaming live sports, gaming, online learning, interactive video applications, and others.
Developing the Technology for Low-Latency Streaming
The delays in conventional live OTT streaming technologies such as HLS and DASH are much longer. They’re caused by relatively long segments (4-10 seconds) and a segment-based delivery model, requiring complete delivery of each media segment before playback. Combined with the buffering strategies used by the HLS or DASH streaming clients, this typically produces delays of 10-30 seconds, or even longer.
To combat delays, recent evolutions of HLS and DASH standards, known as Low-Latency HLS (LL-HLS) and Low-Latency DASH (LL-DASH) , have introduced two key tools:

Chunked video encoding. This is an encoding strategy that produces video segments structured as sequences of much shorter sub-segments or chunks.
Chunked segment transfer. This is an HTTP transfer mode that enables transmission of shorter video chunks to streaming clients as soon as they are generated.

A streaming client can join a low-latency live stream from any Stream Access Point (SAP) which is made available by the live transcoder on segment boundaries or chunk boundaries. Once joined, a player would only need to buffer the latest video chunk generated by the encoder before decoding and rendering it. Considering that each segment can be split into several chunks (typically 4-10), this reduces the delay significantly.
Several open-source and proprietary implementations of low-latency servers and players are currently available on the market, including one by Brightcove. Many of them have demonstrated lower streaming delay when only a single-bitrate stream is used and when they stream over high-speed network connections. However, their performance under more complex and realistic deployment environments has not been well studied.
Testing the Performance of Low-Latency Streaming
In our paper published in ACM Mile-High-Video 2023 (link forthcoming) as well as a presentation at Facebook@Scale , we reported some results after testing the performance of low-latency systems.
First, we developed an evaluation testbed for both LL-HLS and LL-DASH systems. We then evaluated various low-latency players using this testbed, including Dash.js, HLS.js, Shaka player, and Theo player. We also evaluated several of the latest bitrate adaptation algorithms with optimizations for low-latency live players. The evaluation was based on a series of live streaming experiments. These experiments were repeated using identical video content, encoding profiles, and network conditions (emulated by using traces of real-world networks).
Table 1 shows our live video encoding profiles for generating the multi-bitrates, low-latency DASH/HLS stream.

.tg {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-amwm{font-weight:bold;text-align:center;vertical-align:top}

Rendition Video resolution (pixels) Video codec and profile Bitrate (kbps)
low 768 x 432 H.264 main 949
mid 1024 x 576 H.264 main 1854
high 1600 x 900 H.264 main 3624
top 1920 x 1080 H.264 main 5166

Table 1. Live video encoding profiles

Figure 1 shows the workflows of our low-latency DASH and HLS streaming testbeds.

Figure 1. System setup used for evaluation of low-latency players.

Figure 2 shows the available network bandwidth of our emulated LTE mobile network. The download bandwidth of the low-latency players is controlled by our network emulation tool and the network traces.

Figure 2. Available bandwidth of emulated LTE mobile networks (Verizon and T-Mobile)

A variety of system performance metrics (average stream bitrate, amount of downloaded media data, streaming latency) as well as buffering and stream switching statistics were captured and reported in our experiments.
These results have been subsequently used to describe the observed differences in the performance of LL-HLS and LL-DASH players and systems.
A few plots from our study can be seen in the figures below.

Figure 3. Dynamics of bitrate switches reported by LL-HLS and LL-DASH players.

Figure 4. Comparison of variations of latency reported by LL-HLS and LL-DASH players.

Performance statistics – T-Mobile LTE network

.tg {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-1wig{font-weight:bold;text-align:left;vertical-align:top}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-amwm{font-weight:bold;text-align:center;vertical-align:top}
.tg .tg-0lax{text-align:left;vertical-align:top}

Player/Algorithm Avg. bitrate [kbps] Avg. height [pixels] Avg. latency [secs] Latency
var. [secs]
Speed var. [%] Number of switches Buffer events Buffer ratio [%] MBs loaded Objects loaded
DASH.js default 2770 726 3.06 0.21 10.4 93 38 7.99 352.2 256
DASH.js LolP 3496 853 5.65 4.59 22.7 70 53 21.96 369.4 210
DASH.js L2all 3699 908 4.14 3.18 19.9 5 19 7.99 368 147
Shaka player (dash) 3818 916 4.92 2.06 0 16 5 4.66 360.3 155
THEO player (dash) 4594 993 6.16 0.01 0 27 0 0 418.7 152
HLS.js default 2020 1763 562 10.08 10.91 8.1 26 2 9.8 130.7 589
HLS.js LolP 2020 1756 560 5.97 0.2 6.1 24 0 0 148.1 688
HLS.js L2all 2020 1752 560 6 0.23 5.9 34 0 0 133.1 686
HLS.js default 2023 3971 895 8.93 1.13 0 8 0 0 360.8 613
Shaka player (HLS) 3955 908 7.18 2.23 0 14 7 3.8 230 475

Table 2. Performance statistics – T-Mobile LTE network

Evaluating the Performance of Low-Latency Streaming
While LL-HLS and LL-DASH worked well in unconstrained network environments, they struggled in low bandwidth or highly variable networks (which are typical in mobile deployments). Observed effects included highly variable delays, inability to prevent buffering, and frequent bandwidth switches or inability to use the available network bandwidth. Some players simply switched into non-low-latency streaming under such challenging network conditions.
As promising as LL-HLS or LL-DASH are in theory, there is still some room for these technologies to mature.
At Brightcove, we are continuing to work on best-in-class implementations of algorithms for low-latency streaming clients as well as encoder and server-side optimizations. We intend to make our support for low-latency streaming highly scalable, reliable, and fully ready for prime time.
This blog was originally written by Yuriy Reznik in 2021 and has been updated for accuracy and comprehensiveness.

To view our Partner blog, click here