Cloud technology is making itself at home in the broadcast sector, and not just in post-production or distribution, but increasingly for live production and playout. Flexibility, scalability and cost efficiency await the organizations that adopt these solutions, particularly when extra processing capacity is needed above the normal levels compared to building extra capacity in-house.

However, it remains unlikely that all production resources will find their way into cloud infrastructure. For example, running all broadcast processes is likely to work out more costly than owning some of the resources.

So where does this leave broadcasters? In practice, a mixture of on-premises (ground) and cloud production setups allows adopters to benefit from the best of both worlds. For example, everyday production and crisis-situations broadcasting might be handled by on-prem equipment, while cloud would handle peaks above normal usage, such as special events.

But the adoption of hybrid production raises questions as to how workflows spanning resources from on-premises to the cloud are effectively managed—otherwise production staff may be drawn away from focusing on delivering engaging content. Management of these workflows, or orchestration, is therefore essential.

Framing Orchestration for a Hybrid Future
Whether it’s live production in the cloud, outsourcing part of an on-prem production into the cloud as a hybrid workflow, or delivery of live content to a pop-up playout channel, they all require control over on-premises and cloud resources. They need to be connected with each other to ensure the safe and reliable flow of media.

Controlling and orchestrating connectivity and signal routing in the cloud itself can often be handled by established static-events automation tools. For example, predefined automation templates can be quickly deployed to create cloud infrastructure. But this approach does not extend to on-premises resource control, which is often essential for larger live productions or permanent workflows.

Orchestration systems for hybrid live production must therefore integrate both on-premises and cloud resources seamlessly. The goal should be for these workflows to be managed in the same way as a traditional master control room or production gallery, with end-to-end visibility, monitoring, and efficient management of both on-premises resources such as bandwidth, ports and equipment, and cloud instances.

Must-Have Features for Orchestration Systems
In hybrid live production setups, operators should simply be able to select a source and a receiver, whether they are based on-premises or in the cloud. The orchestration system should then be able to automatically allocate the required encoders, ARQ protection and cloud gateways from a managed pool of resources.

For example, choosing the right compression technology is often a balance between quality, bandwidth and latency, and configuration might cover codec parameters, encryption and network addressing.

All of these decisions should be made by the system in the background, ensuring that users aren’t presented with unnecessary complexity. To make the management process even more seamless for staff, an intuitive control surface, such as a familiar hardware panel or an easy-to-use software interface, would allow them to keep track of on-premises, ground-to-cloud-cloud-to-ground (GCCG) and cloud workflows.

A hybrid connection may cover a number of networks, including the on-premises production network, dedicated WAN links and unmanaged networks such as the internet. An orchestration system must monitor these network types and provide redundancy in the event of failure. It also needs to handle multi-vendor equipment, including IP switches, cameras, switchers, servers and multiviewers, to ensure end-to-end control. Transcoding between on-premises feeds and cloud receivers is also essential, alongside orchestration of the connectivity to the cloud and between cloud-native applications.

In order to make the best use of the pay-per-use model offered by a cloud solution, the orchestration system should be able to interface with public cloud providers and automate the scaling of cloud instances by spinning them up or down. If a connection is no longer needed, automatic termination of cloud processing instances can save on hosting costs. Where spinning instances up and down isn’t practical, such as for larger instances in a cloud-based video switcher, starting and stopping instances can also reduce costs, although typically not as much as the spinning option.

And just as importantly, no orchestration system should be lacking in effective security measures. While public clouds usually offer enterprise-grade security, the system must cover secure communication between on-premises devices and cloud instances. It can do this via encrypted communication protocols, role-based access control (RBAC), integration with identity and access management systems via Single Sign-On, ARQ stream encryption management and audit logging for full traceability of operations.

The Importance of an Integrated Orchestration Approach
As live production evolves, the broadcast industry is embracing hybrid models that blend the strengths of on-premises infrastructure with the flexibility of cloud. But to unlock the full value of this approach, orchestration must take center stage to provide the operational efficiency, cost control, and resilience. Systems that can intelligently and seamlessly manage resources across both environments will define the future of live production.

Looking ahead, AI could enhance the automation and efficiency of orchestration systems, automating complex workflows, reducing manual intervention, and minimizing human error. With dynamic scaling of resources also an AI-led possibility, organizations will be further empowered to make a success of hybrid live production and continue to create high-quality content for audiences.

In recent years, during the move to IP in broadcast production, broadcasters have predominately focused on video transport due to it requiring so much bandwidth. However, audio also presents challenges. Compared to video, not only does audio involve a substantially greater number of flows, but it also uses a diverse number of standards in production.

Professional audio over Ethernet has been around since before the turn of the century, with broadcast radio being an early adopter of standardized network technology. Over the years, several competing proprietary approaches and standards for audio over IP have emerged, including DANTE, REVENA MADI (AES10) and AES67. However, the compatibility between the various approaches and even between implementations of specific formats has been a long-standing issue in audio transport and processing. With an increasing number of broadcasters moving to IP in the facilities, this issue of ensuring audio compatibility has become critical.

As broadcasters look to overcome the complexities presented by audio in the move to IP, they must consider the following key issues:

THE STREAMING PLANE

The streaming plane refers to the basic transport of the audio over the network. In that context, AES67 has become key. First issued in 2013, the AES67 standard has been adopted and integrated by most manufacturers, including providers of products based on proprietary approaches. Crucially, AES67 is also the basis of the recent SMPTE ST 2110-30 standard, which means that compatibility on the streaming plane between most of the popular solutions is largely assured.

That said, within the SMPTE ST 2110-30 standard, three levels of conformance are defined—only some of which are currently supported by vendors. The mandatory Level A provides support for 48 kHz streams with one to eight audio channels, at packet times of 1 ms. Level B adds support for packet times of 125 µs. Level C increases the max number of audio channels allowed per stream to 64. The latter means that MADI, which continues to enjoy a lot of popularity, may be carried as-is over the audio network.

What broadcasters often fail to note as they consider their move to IP is that many audio-over-IP systems are currently only able to handle the basic level A. They may also have limitations when it comes to the total number of audio network streams supported, and what combinations of channel count and stream count can be used. So, while the manufacturers can genuinely claim support for SMPTE ST 2110-30, the limited scope of their compliance should be taken into careful consideration when selecting audio equipment as they could place a restriction on the flexibility of the overall workflow.

TIMING

As part of implementing AES67 compatibility, Precision Time Protocol (PTP) version 2, or IEEE 1588-2008, can now be used for timing of the network by the different manufacturers. This also fits with the SMPTE ST 2110-10 standard that mandates use of PTP v2. SMPTE has also published the ST-2059 standard, which generalizes the media clock concept of AES67 to any kind of periodic media clock, including video and timecode.

CONTROL PLANE

The common production environment has many more audio sources than video and an even greater number of destinations. A major sports production could have thousands of audio channels travelling across the network, for example. So, while audio may not necessarily place high demands on bandwidth in an IP network compared to video, it certainly creates a challenge in terms of control and orchestration.

Audio engineers expect to be able to “plug and play” equipment and connect sources and destinations without concerns about protocols and standards. On the other hand, in a broadcast facility, inter-studio routing must be centrally controlled both for the integrity of signals, but also for security and access control.

The apparent strength of some of the proprietary approaches is that they include a comprehensive control plane, whereas standards like AES67 or indeed SMPTE ST-2110 do not define how the streams should be controlled.

PROPRIETARY APPROACHES

While the proprietary control planes are effective on their own, they are not compatible with each other. More crucially, they are designed for a local studio environment (LAN) and therefore aren’t suited to a seamless distributed production environment, such as for big campus or inter-campus use, or for remote production (over WAN).

Furthermore, these control planes rely on audio being made seamlessly available to any equipment in the network by default, meaning no explicit routing of streams is required. This could be a security concern, especially in a distributed, multidepartment or multi-organization environment.

In addition, the fundamental assumption behind this approach that no controlled bandwidth management is needed may be flawed when the size and complexity of the network increases.

MADI TIELINES

One approach to overcoming the issues with control plane interoperability, and address security and stability concerns, is to bridge different IP audio “islands” by using MADI baseband tielines. However, this adds complexity to the management of audio routing in the campus and reduces flexibility and agility. Essentially, this approach largely defeats the purpose and promise of using a converged media network in the first place.

NMOS

The Networked Media Open Specifications (NMOS), developed by the Advanced Media Workflow Association (AMWA), offers a way to address endpoint control for audio in a way that may deliver the true promise of distributed IP production. The standard is now gaining traction in the industry—although its uptake among audio equipment manufacturers is lagging behind that of video equipment vendors.

SDN

In the meantime, the most promising approach to control audio flows in an IP network is to use software defined networking (SDN). This not only provides an easy way to connect diverse sources and destinations, but it also adds a layer of predictability, performance guarantees and security by managing bandwidth and only allowing authorized destinations access to specific audio network flows.

PROTECTION

As production transitions from the LAN environment and into the WAN, and IP audio networking is converged with video networking, audio signal protection is becoming an issue. The SMPTE ST 2022-7 dual path protection standard has now been extended beyond video—to cover any RTP media stream—and provides a great way to ensure audio signal reliability. There may still be compatibility and network addressing issues where different parties need to exchange audio signals, e.g. between different organizations, or simply between an OB van and the live audio system. Broadcasters can address these concerns through IP Media Edge devices and/or SDN controlling which flows can cross the boundary and how—a better approach than using MADI tielines to bridge the gap.

WHAT’S NEXT FOR AUDIO?

The move to audio over IP will make it easier for broadcasters to keep up with the latest developments. For example, immersive audio authoring typically requires up to 127 dedicated audio object channels, in addition to a base surround signal containing up to 22+2 audio channels. Traditionally MADI has been used to interface this high channel count in the production stage, but IP networked audio has higher capacity and can fit all these channels in one cable. Audio over IP is also more flexible with regards to routing and does not require any expensive and dedicated MADI routers when more complex topologies than point-to-point links are required. The use of audio over IP means there is less need for dedicated or custom hardware, allowing for virtualized and flexible workflows. With increased capabilities to adopt new trends such as immersive audio, broadcasters will be able to offer an enhanced viewing experience to audiences.

Olivier Suard is the vice president of Marketing for Nevion.

For a long time, professional-grade, real-time media transport, processing and monitoring functionality was provided by dedicated hardware. If additional functionality was required in the network, a “box” would be purchased that specifically performed that task.

This dedicated hardware approach made technical sense because of the amount of processing required, as well as the need for low latency, high reliability and minimum power consumption. However, this was inflexible and not always cost effective.

The trend for high-tech equipment in many industries has been a progressive move from dedicated hardware to specialized platforms running software, and then onto COTS (Commercial Off-The-Shelf) IT hardware, such as servers based on X86 CPU architecture, running software applications. For real-time transport, processing and monitoring of media signals, the concept of software-defined platforms is now becoming established. The question is though: how suitable is COTS for live broadcast production?

Intuitively, a COTS approach makes sense, as IT equipment is ubiquitous, open and proven in many environments. Despite some initial hesitancy in the industry, we are beginning to see media processing products appearing that are based on COTS.

OVERCOMING THE CHALLENGES

Currently, COTS products for live media transport, processing and monitoring still only provide limited functionality compared to leading software defined platforms. Before it is possible for COTS hardware to be used for all aspects of live production there are several technical challenges that the industry must firstly work to overcome.

But what exactly are these challenges and what could overcoming them mean for the role of COTS hardware in live production in the future?

Performance

At this stage, the main obstacle to using COTS hardware is its real-time processing performance. While it can handle applications—such as audio processing—that involve flows under 10Mbs reasonably well, it can be less efficient with higher rate flows, such as those required for real-time video encoding.

However, developments are taking place that will help running video processing applications on COTS platforms. Firstly, we are beginning to see newer generations of standards, such as JPEG XS encoding, that have been designed from the ground-up for software processing. Secondly, and more fundamentally, chip vendors, such as Intel and Xilinx, are launching or already offer new “generic” FPGA acceleration boards.

The NICs (Network Interface Cards) used for stream acquisition can also be a processing bottleneck. To get the performance needed to handle very high volumes of data in real-time, it helps if the NICs can communicate directly with the FPGA, rather than via the CPU and memory. Vendors such as Mellanox have developed NICs designed to optimize the throughput to the processing functions.

It is worth noting though that a contributing factor to some of the perceived inferior performance on COTS is poor programming; for example, it is too easy to add buffering to solve problems rather than address them in a way that keeps latency down. Lean and efficient code is key to powerful and scalable media functions implementation.

Packet Pacing

Traditionally, professional media transport has been linear in nature, meaning that the output of equipment utilizes a constant, defined bit rate. In IP terms, this means that packets are transmitted at a constant and steady pace.

COTS equipment is inherently nonlinear, which is a challenge to broadcast orthodoxy. Typically, the software and hardware—due to the resource scheduler—will natively try and push out as much data onto the network as quickly as possible, without any consideration for pacing.

The problem with nonlinear transmission is that bandwidth usage becomes very unpredictable and timing consistency cannot be maintained. And in real-time applications, where packet delays are unacceptable, there should always be enough bandwidth to accommodate all possible flows and keep an accurate packet pace.

The nonlinearity of COTS devices creates the need for substantial additional bandwidth, which is largely unused most of the time and is not desirable economically.

Consequently, the prevailing view in the industry is that COTS-based media applications should be developed in a way that paces packets more carefully.

Power Consumption

CPU-based COTS devices have noticeably higher power consumption than bespoke hardware or software-defined platforms based on FPGAs. In some production environments, such as OB vans or remote sites, this can prove to be a challenge.

Over time, COTS power consumption is likely to be reduced, as generic systems introduce FPGA acceleration.

It is also likely (though not necessarily desirable) that broadcast environments will become more accommodating of the extra power needed for COTS.

DOES THE FUTURE LIE IN COTS?

There is no doubt that, for most real-time broadcast media transport, processing and monitoring, the best approach currently is to use software-defined platforms, built on hardware optimized for performance.

While truly generic COTS IT hardware running software is not yet a viable option for many applications—especially video processing that requires very high bandwidth—it is already being used successfully for some applications. Therefore, as technology continues to evolve and is capable of overcoming the challenges and limitations outlined above, it is likely that COTS will be suitable for use in all aspects of real-time broadcast production.

Olivier Suard is vice president of marketing for Nevion.

The search for increased productivity pushes many broadcasters towards IP technology. Considered less expensive than broadcast-specific baseband, one of its main attractions is its ability to handle any video and audio technology.

In many cases though, broadcasters initially opt for a “like-for-like” network replacement. This misses the key point that IP brings with it the so-called “IP LAN/WAN convergence,” which makes it easier to achieve savings and increased flexibility by sharing equipment, studios, control rooms and production staff across locations.

Many broadcasters misguidedly turn to poorly-informed IP switch vendors for advice, when they need an approach that accommodates the specific needs of broadcasting. For that reason, the initial focus should be on getting the network architecture and the control right, which means considering the three main architecture layouts that are typically used.

CENTRALIZED STAR NETWORK

In architecture terms, the tendency for most broadcasters is to adopt what is known as a “centralized star network” with all connections transiting through a large IP router that can be located in the master control room.

The main disadvantage is that everything needs to travel to the central router, requiring expensive fiber connections with every single device.

[Read: IP Is Changing The Future Of The Broadcast Control Room]

Scalability is another problem. Capacity is often reached sooner than anticipated, necessitating replacement of the central router. Since every connected device occupies one expensive high-bandwidth port on the central router, the cost-per-port for low bandwidth devices is high.

Moreover, lack of aggregation means redundancy needs to be handled by edge devices, necessitating two connections to the central router, or one to each of the main and backup switches. Finally, the assumption that all traffic will transit through the central router makes star network architecture unsuitable for treating remote locations as extensions of the main location.

SPINE LEAF

The second model is “spine-leaf architecture,” involving two or more routers at the core (spine) and other smaller routers at the edge (leaf).

This reduces the number of connections going directly to the main routers, leading to simplified fiber management. It requires fewer ports on the central router(s), and delivers more effective cost-per-port, especially for low-bandwidth devices.

Spine-leaf architecture reduces the cost of building-in redundancy and also provides optimal flexibility and scalability. Networks no longer have to be oversized from the outset, since capacity can be added over time.

While true spine-leaf architecture can be more complex than other approaches, it is a scalable, resilient and high-performance structure perfectly suited to the needs of broadcasters.

DUAL STAR

This third architecture model is “dual star” architecture which still involves the use of two spine routers, but with the difference that each leaf in the network is only connected to one spine.

Unfortunately, this is not a flexible approach when it comes to load distribution and optimization of total network capacity. As a “pseudo” spine-leaf approach, it puts special requirements on end devices that need redundant connections when the network evolves and it also suffers from redundancy problems.

The proponents of this architecture usually favor automatic protocol-based routing rather than software-defined networking. Yet, despite being better suited to automated routing, the dual star is not a preferable option overall. Only a true spine-leaf architecture enables broadcasters to get the most out of their IP infrastructure investment in their facilities.

However, as well as network architecture, broadcasters also need to orchestrate and control the IP media network, weighing up the comparative advantages of automatic routing and SDN.

AUTOMATIC ROUTING

Automatic routing means leaving the decision about how to transport individual media flows to the network, rather than the operator.

While automatic routing — and the IGMP and PIM protocols — are used widely in IP networks across the world, they have disadvantages in terms of performance and bandwidth management.

Automatic routing may not be fast enough for live production and can run into trouble with network loops. This can only be fixed through higher operational complexity — and unless care is taken in designing and controlling the network there is a risk of oversubscribing it, causing instability and signal drop-outs. On top of this, there are also concerns around protection and security, as streams to destinations are not explicitly controlled.

SOFTWARE-DEFINED NETWORK ROUTING

SDN puts routing control in the hands of a centralized control layer. The management and orchestration software holds a complete view of the available equipment, the network infrastructure and the services. This enables it to make intelligent decisions on routing and controlling flows and provides the explicit routing capability that broadcasters expect and need.

This has many advantages. Firstly, it guarantees a higher level of performance when compared with automatic routing, since the software is also in control of every media flow. It is even beneficial from a protection and security perspective as the orchestration and control software can easily create path diversity to protect failures and can also reduce security risks by fully controlling which destination is allowed to receive which multicast.

Unlike automated routing, SDN can, with the right software, easily handle any network architecture. Ultimately, all its advantages make it the control of choice for the creation of truly flexible, scalable and high-performance IP media networks.

Despite the obvious benefits of IP technology, broadcasters should bear in mind that a successful IP infrastructure is built around the “ground-up design” of infrastructure. Success is also largely determined by the way in which individual elements within the network are controlled.

Broadcasters should be architecting a network using a true spine-leaf model, and controlling the elements within it using SDN routing. It is the most effective way of maximizing IP technology, in turn helping deliver optimal return on investment and higher chances of operational success.

Olivier Suard is vice president of marketing for Nevion.

[Want more information like this? Subscribe to our newsletter and get it delivered right to your inbox.]