One of the early problems with live video streaming—first demonstrated at the 1997 NAB Show—was that the images hiccupped and sputtered due to constraints in bandwidth. Yet, even with those annoying imperfections, it was clear from day one that video had a future over the internet. 

Now—25 years later—getting rid of those video interruptions over the net is close to reality. It hasn’t been an easy engineering problem to solve. In live video streaming, it is important to remember that the lower the latency, the less robust the video signal becomes. Low latency video can hiccup, or stop playing completely, after even the tiniest gap in the bandwidth of the stream.

Balancing Act
Since day one, live video streaming has been a balancing act—low latency versus stability of the content stream. (Though things have dramatically improved from the days when virtually every online video was shaky and dropped out frequently.)

Today, depending on the value of the video content, latency takes on varying degrees of importance. With the standard streamed church service or town council meeting, which ranks as the lowest-valued content, segment size can be reduced from 2-4 seconds with little cost or concern. In this class of video, latency can be reduced to between 6–12 seconds.

More valuable, but pre-recorded, broadcast content averages about 5-6 seconds of latency, while demanding real-time content like OTT streaming of live online sports, gambling or gaming demand special low-latency technology. New technologies with complex names are dealing with the latency issue and programming vendors are expected to charge a premium for the service.

Until recently, ultra-low latencies of less than a second were best achieved using UDP-based WebRTC (Web Real-Time Communication), a free and open source protocol introduced by Google in 2011. It provides web browsers and mobile applications with real time communication via program interfaces. 

Until about 2020, the HTTP approach—a competing method—could not provide a low enough latency for interactivity, so WebRTC remained a popular solution. 

Apple vs. Microsoft (Again)
However, the cost and complexity of encoding and storing the same video file twice for the two main computer platforms—Apple and Microsoft—made processing and storage of content expensive. Two versions of the same video stream had to be made either in advance or instantly. With users accessing streams across iPhones, smart TVs, Xboxes and PCs, this expensive complexity became a major issue.

In 2016, Apple and Microsoft suggested that the Moving Pictures Expert Group (MPEG) create a new uniform standard called “Common Media Application Format” to simplify online delivery of HTTP-based streaming media (Fig. 1). It was published in 2018. 

Once the standard was created, manufacturers implemented it quickly. The benefits of encoding, packaging and caching a single container for video delivery was obvious. But CMAF did more than just reduce encoding complexity.

HTTP-based video delivery still lacked the real-time delivery options that viewers wanted. CMAF had to also improve latency. Now, Microsoft and Apple have agreed to reach audiences across the HLS and DASH protocols by using CMAF, a standardized transport container (Fig. 2). 

CMAF represents a coordinated industry-wide effort to lower latency with chunked encoding and transfer encoding. It also supports file encryption and digital rights management. Multiple incompatible DRMs are supported, including FairPlay, PlayReady and Widevine.

Since its creation, CMAF has opened the way for a new generation of low latency technology. These consist of Low Latency HLS (LL-HLS), Low Latency DASH (LL-DASH) and the High Efficiency Stream Protocol (HESP). CMAF works hand-in-hand with these protocols.

LL-HLS, which was announced by Apple in 2020, became the most used technology for streams via HLS. There is a DVB standard for low-latency DASH and there is ongoing work to ensure interoperability for all DASH/CMAF low-latency applications. 

Harmonic and Akamai showed low-latency CMAF demos with a latency of under 5 seconds at the 2017 NAB and IBC shows. Since then, most other encoder and player vendors have integrated the technologies into their products. 

Market Competition
All low-latency systems work in the same basic way. Rather than waiting until a complete segment is encoded—which usually takes between six and ten seconds—the encoder creates much shorter “chunks” that are transferred to the content delivery network as soon as they are complete.

In a new forecast from Rethink TV, a research organization, it is predicted that low latency protocols will quickly find a place in live sports delivery. As time goes on, all content—not just sports—will make use of the protocols, the researcher predicted.

Rethink TV anticipates low-latency delivery will be a very competitive market in the next few years. These low-latency distribution technologies, which can be offered as a premium service to subscribers, are key for a number of competitive video technology vendors and users. Pivotal will be knowing how to optimize both the centralized and edge infrastructure to compete with rivals, the research found. 

As most devices are now totally compatible with CMAF, Rethink TV predicts most will be able to receive video streams packaged in LL-HLS and LL-DASH formats. Competitive forces will drive adoption.

For owners of content, rights holders and streaming services, the reduction of latency for live video can differentiate them from their rivals by dramatically improving the reliability of video service.

This also applies to broadcasters that operate both traditional pay-TV distribution infrastructure and an OTT service. Those broadcasters need to get their latencies down to something nearer their traditional pay-TV competitors.  

Reducing OTT latency is essential, but now for only live content. It will probably remain in the sports and betting arena for the time being. However, all on-demand video programming will eventually migrate to a low-latency technology, probably before the end of this decade. 

Over the past several years, the video streaming industry has been working on solutions to achieve low-latency OTT delivery in order to compete with traditional broadcast of live content. This was initially achieved with the MPEG DASH format, using CMAF, standardized in January 2018 by MPEG, followed in October 2019 by DVB delivering the low latency update of its DVB DASH specification. In June 2019, Apple announced during its Worldwide Developers Conference that the HLS protocol is being updated to support low latency, (the “Apple way,” of course).

The challenge: Two separate ecosystems exist today for the delivery of live content in DASH and HLS, the most popular delivery formats. In order for OTT to scale, we have to rationalize the two approaches and find a common groundwork for low latency.

CMAF: A FOUNDATION FOR CONVERGENCE

As a solution to possible convergence, Common Media Application format (CMAF), an ISO media format standardized in January 2018, provides a promising framework. When Microsoft, Apple and other industry players came together to propose the CMAF standard, many believed convergence was a done deal.

But this was not and is still not the case, and there are still some missing pieces to the puzzle. Service providers that wish to distribute their content in both formats (i.e., HLS and DASH) still need to have separate workflows at some points and duplication of media file caching in the network, which increases their overall costs; it also increases complexity and puts more stress on the OTT business model and therefore slows down OTT deployments.

WHY ISN'T THERE A SINGLE WORKFLOW FOR LIVE CONTENT DELIVERY?

Low-latency delivery is an essential feature for highly valuable content such as premium sports, which also drags the largest audience for live. Therefore, now more than ever, it is important for the industry to figure out a way to produce and cache a single set of media files in the network to optimize processing, caching and storage costs. This was the promise of CMAF from its onset, but hurdles with regards to low latency delivery methods need to be removed to make it a reality.

Indeed Low Latency DASH, also called “DASH LLC” (Low latency Chunk), and Low latency HLS, as proposed by Apple in June 2019 (LL-HLS), rely on different sets of tools and protocols for delivery, despite both using CMAF (including CMAF chunks) for the media file format. The former uses HTTP chunk transfer encoding while the latter (which is still an Apple draft specification) uses HTTP/2 and the more problematic HTTP/2 push method (not widely deployed or even supported), creating a lot of pushback from the CDN industry.

As a result, for the delivery of low latency services, the OTT industry is still grappling with two sets of media files, duplicate caching and high costs while fighting to find a profitable business model.

THE LATEST UPDATE ON LOW LATENCY

The end of 2019 brought about positive changes to the OTT industry. Apple and other industry players, including technology solution providers for encoding, packaging, origin, CDN and OTT players, expressed the willingness to find a compatible mode between DASH and HLS for low-latency content delivery.

It would be beneficial for the industry to see some evolutions on the current draft LL-HLS proposal to finally achieve the initial promise of having a real common format for media files and a common delivery workflow.

CONTENT PROTECTION: DRM AND AD INSERTION

For service providers, content needs to be monetized, so DRM and ad insertion tools are therefore critical to ensuring that service providers can take advantage of the abundant revenue opportunities in the OTT environment. As part of the global workflows for content delivery, there should also be a way to build common methods for these features.

There is now unified support for the CBCS encryption scheme by all of the major DRM systems such as Apple FairPlay for HLS, and Google Widevine and Microsoft PlayReady for DASH. With this approach a single media file can be encrypted in the CBCS format and distributed to any player, no matter what DRM system is in use. This fulfills the initial promise of CMAF. Detailed discussions still need to take place around DRM, but the industry is headed in the right direction.

Ad insertion—in particular dynamic and targeted ad insertion—is increasingly being used by OTT service providers to make ads more relevant. Through dynamic ad insertion solutions, service providers can dramatically boost their OTT revenues by creating more value for advertisers and improving the end-user experience.

For targeted ads to be successful, service providers need to ensure a seamless video experience, with smooth dynamic ad insertion. Both DASH and HLS Low Latency specifications are currently scrutinized to make sure the industry practices used in legacy workflows can be used or slightly adapted to take into account the additional constraints brought about by the shorter delivery path. Guidelines for both HLS and DASH will have to be updated to properly use the timing information for inserting ads in the low latency streams. It is still a work in progress but there is hope this can be achieved to enable a common workflow.

CONCLUSION

At Harmonic, we believe that the industry has to develop a unified CMAF based, Live OTT streaming system that supports: common encryption (already done with CBCS), ad insertion and low latency for both delivery formats (HLS and DASH).

Under this approach, media files are only stored once, which saves processing, storage and delivery costs. And, of course, there are two different HLS and DASH manifests that can always be cached separately.

This will create an optimized delivery scheme that will enable all of the key features OTT operators are looking for. Failing to accomplish this will fragment the market, make OTT less profitable and slow down its development.

Patrick Gendron is director of innovation at Harmonic.

Go back a couple of decades in a time machine, and the Real Time Media Protocol (RTMP) was the protocol for streaming media. Video encoders pushed out RTMP for delivery to the cloud and, in turn, to a player. Everyone relied on RTMP, and the workflow was simple, without complications such as cloud transcoding or repacking.

Fast-forward to 2019 and RTMP is still widely used, although it doesn’t support High Efficiency Video Coding (HEVC, or H.265) and no longer has native support within mainstream player applications and operating systems. The industry continues to rely on RTMP-based workflows with streams pushed up to the cloud and then transcoded, transrated and repackaged before they are ready for distribution in the formats that today’s viewing devices support.

BEST OF BOTH WORLDS

However, a new encoding and streaming model is gaining ground—and earning support as a smarter and more cost-effective means of modern media delivery. The combination of HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) with new chip-level system hardware, along with the concept of edge compute encoding, is making the entire media delivery process faster and more efficient. And, HLS and DASH combined with the Common Media Application Framework (CMAF) enables low latency.

As a result, video delivery over the public internet is being achieved with latency levels low enough to enable a new set of applications—for example, increased interactivity that allows channel changing in emerging countries (where cable is not feasible) to be made possible. Other applications can include live auctions and sports gamification—and making streaming of “live broadcasts” actually live.

While neither HLS nor DASH is new, what is new is CMAF and the development of cost-effective live streaming video encoders that natively support HLS, DASH and CMAF. This advancement in technology is thanks to new system-on-chip (SoC) semiconductor technology that has allowed video encoders to now use more modern, purpose-built hardware rather than general purpose FPGAs and CPUs.

Built on the Qualcomm Snapdragon SoC, new edge compute encoding devices boast significantly greater processing power than their legacy counterparts. With this additional horsepower, edge compute encoding devices can address two critical challenges in streaming: adapting smoothly to changing network conditions and delivering high-quality video over relatively poor internet connections.

Content providers use adaptive bitrate (ABR) streaming to provide every viewer with the best possible image quality. With anywhere from four to eight versions of content—all with different resolutions and bit rates—available in the cloud, it’s possible to deliver the version that takes best advantage of available bandwidth for the target device. One version might be ideal for a small cellphone picture; one might be intended for an iPad, another for a PC or a set-top box.

BRINGING LIVE STREAMING TO INTERNET-CHALLENGED AREAS

The limitations of previous-generation encoding devices made it necessary to create all bit-rate variants of that initial piece of content in the cloud. Why? Because to take on the processing required to create multiple versions on the point of production, conventional encoders would need to be equipped with a bigger CPU or with a lot more FPGAs, and the costs for those add up. As mentioned, in a traditional streaming scenario, higher-resolution content is pushed at a particular data rate—often about 10 Mbps—up to the cloud, and then cloud processing outputs versions with different resolutions and bit rates.

In the new edge compute encoding model, the SoC-based device creates multiple versions of content and delivers them to the cloud for distribution. The latest generation of SoC-powered edge compute encoders can create six instances of video at different bit rates and send them all to the cloud, reducing both latency and the cost of cloud processing while maintaining image quality. And new SoC-based devices can create both HLS and DASH outputs, operating within CMAF, carrying HEVC to the cloud in a direct workflow that allows users to enjoy the bandwidth savings—upwards of 40% depending on the content—that HEVC affords.

Thanks to the high compression and high quality of HEVC, 4K video can be encoded and sent up to the cloud, thereby boosting the video quality offered to customers. While HEVC enables 4K video over public internet, consider the advantages of HEVC to a 720p signal. Maintaining outstanding quality of the 720p signal with less than 1 Mbps is now possible, opening internet-challenged areas for live streaming. Even commodity internet routing and switching gear available at consumer retail stores can be used to produce best-in-class video. This is the democratization of technology through the use of today’s standards.

The application of SoC-based edge compute encoding to streaming is transforming media-delivery workflows and the cost structures associated with them. Even more exciting is the prospect of the new applications that edge compute encoding can enable and engender. If anyone can take a camera, plug it into an encoding device and stream video worldwide in 3 seconds or less, what happens then? The possibilities are limitless, and the results will be amazing.

Todd Erdley is the CEO of Videon.