Scalable automation

Opening doors to new distribution opportunities

The ability to reach a more segmented market through the delivery of specific media has driven broadcasters to increase the number of distribution channels they operate in search of new sources of revenue. Digital content management technology makes media delivery and consumption possible through an expanding variety of channels, which reach a larger number of diverse audiences or market segments.

According to PriceWaterhouseCoopers' “Global Entertainment and Media Outlook: 2006-2010,” world advertising revenue grows 6 percent each year. Internet-based content distribution companies outpace that, with ad revenues increasing 18 percent annually. Television is maintaining its advertising market share, but must develop loyalty across alternative delivery channels to sustain revenue levels. The competition to increase revenues is a race to reach the more diversified and discriminating audiences.

As media distribution expands to multiple content delivery formats, more granular market segments and an increasingly diverse set of delivery devices, playout automation must deliver greater reliability, expandability and application interoperability. Media distribution suppliers must provide solutions that address both business and technical operations. This would enable broadcasters to manage growth, support expansion with new advertising and subscriptions sales, and deliver content streams in a manner that provides the perfect audience experience across every channel.

Broadcasters and other media distributors demand that playout automation solutions enhance their reputation and brand image. In addition, they expect resulting workflow environments to reduce labor requirements and constrain costs.

Manufacturers are responding with solutions that enable expanding media distributors to reliably scale their automation environment and seamlessly incorporate automation with traffic, asset management and other applications. This generates flawless content presentation across an increasing number of playout channels, while providing true economies of scale in media delivery operations.

The way forward

Media distributors require an environment characterized by guaranteed frame-accurate switching, reliability in the software and hardware solution elements, historically high system availability statistics and the ability to manage large numbers of channels across a single architecture. There is a further need to combine playout automation into a seamless application environment that enables dramatic expansion of revenue generation capacity, while improving productivity through the automation of information transfers across traffic, asset management, playout automation and other applications.

In a digital world, it is much easier to acquire, archive, schedule, manage and repurpose content for specific audiences than it was when everything was tape-based. Success in the digital world requires building workflows that optimize the flow of file-based rich media.

Despite the clear benefits, transitioning to a digital environment often presents a considerable challenge. Broadcasters moving to a tapeless environment typically have an extensive, valuable tape library and an installed process that supports linear media distribution. This requires them to continue supporting analog operations during their transition to the digital environment. Applications must support the continued ingest, management and playout of tape-based content, as well as the transcoding of linear media to digital formats for future use. They must also provide the capability to ingest, manage and play out digital, file-based content.

System availability: The first principle

Playout automation systems are created with software and hardware elements that must ensure channel playout continuity without degradation. The optimal playout automation system incorporates the following features and functionalities:

  1. Application-specific software coreThe starting point for system assurance is the use of a real-time OS combined with total control of the mission-critical operating environment. Linux 2.6 has a preemptable kernel that provides real-time performance suitable for broadcast applications and delivers the scalability needed for even the largest multichannel applications. These characteristics drive the use of Linux in 80 percent of the world's top 500 supercomputer installations, according to TOP500 stats ( it is an open source OS, a distribution can be provided that incorporates only the resources necessary for the specific application. Any issues that affect the automation system can be fixed by the application development team without relying on a third-party supplier. As a modern operating system, Linux provides full support for current and future hardware devices, software architectures and development tools.
  2. RedundancyDual-redundant component architectures enable quick recovery from failure. Device controllers must be configured to run in a fully mirrored configuration with comprehensive proprietary mirroring techniques used to ensure real-time synchronization and auto-failover. This should be reinforced by a multiple redundant LAN configuration, with automatic switching between LANs in the event of failure. The switching should be handled by the application, not the OS.Figure 1 presents a playout automation environment that exhibits the device redundancy and the LAN duplication required for assured continuity of playout operations on every channel.
  3. Instantaneous operational visibilityProtection against automation failures is just one requirement. The system must also provide tools to prevent and recover from operational and other equipment failures. To ensure that malfunctions are recognized and corrected quickly, a comprehensive error reporting capability is essential. Reporting should provide sufficient time for resolution before playout operations are affected. System-wide alarms and individual errors must be reported whether they are technical or within schedules. The best automation applications will present these and other system faults on an operator screen, associated control panels and external tracking systems.An excellent reporting system will provide the following: Warnings on the live schedule status and schedule fault conditions for all attached devices. Polling of device status, whether devices are included in active schedules or not. This maximizes response time in case of issues. Conflict checking through advance operator notification of pending conflicts between resources. Logging facilities for observation and analysis of both system and operator activity.
  4. System backupThe system should provide multiple backup equipment options in the broadcast chain to fit varying budget constraints: Failover can be automatic (based on device status) or manual. All or part of the broadcast chain can be run fully mirrored for instant failover and protection. Devices can be run in an N+1 or N+M backup configuration for less demanding applications. Individual events can have backup resources scheduled for occasional backup.
  5. Online upgrade and maintenanceOn-air (live) software upgrade capabilities are important to ensure that channels are available during system expansion, upgrade or maintenance. Using a manual transfer option, operational control can be switched between the main and standby systems to accommodate offline enhancements. Transfer between the two systems should be transparent to channel audiences. The capability eliminates the potential for disruptions that could take the system offline.
  1. The new network for automation Implementation of proper network topologies isolates broadcast operations to eliminate effects of non-broadcast traffic and enhance application performance. They provide protection of content essence and content metadata from corruption and misappropriation. Application support for TCP/IP and the Open Systems Interconnection initiative optimizes the network for management of digital, file-based material. Ethernet is ubiquitous in private and public networks that operate at 100Mb/s and 1Gb/s speeds today — and soon will be at 10Gb/s data rates. Routers and switches incorporate the operational capability to segment networks and provide security through the use of access controls, firewalls, VPNs, intrusion prevention/detection and virus protection.

Figure 2 shows an example of a playout automation system incorporating multiple redundant LANs that deliver specialized functionality, isolate other network segments and provide automatic LAN failover.

Scalability: The path to salvation

Ensuring business and playout continuity in the face of a catastrophic event requires special planning. The capability now exists to automatically mirror playout schedules in geographically separated facilities. This capability enables schedules spread across multiple automation systems in one location to be synchronized to schedules on a single or multiple systems in another location. Backup and main schedules are mixed across the sites with the system able to accommodate source and bus remapping and configuration differences between the sites.

Any automation application expecting to support large environments must manage large numbers of playout channels within a single architecture. The most cost-effective way to support new, lower revenue channels is to be able to add them onto an existing control system at minimal cost, allowing existing operation staff to manage them.

The system should allow operations to scale in different ways to meet various preferences. A partitioned architecture for small to medium installations is shown in Figure 3 on page 38. It can scale in the conventional way — supported by many automation systems through the building of multiple pods, around pairs of device controllers sharing a common database/media manager. Through the choice of hardware and use of an interrupt-driven, pre-emptable kernel, each controller pair can support 30-plus actual playout channels.

Alternatively, by using a master- or device-distributed architecture (right side of Figure 3), the system can scale seamlessly to 1000 or more channels with no loss of functionality. Any operator can access any channel if required, and any source can go out on any channel. There is no loss of functionality as the system scales, as typically happens with the “podded” approach.

Seamless application workflows

For example, any channel can be linked to any other channel for regional breakaways on a dynamic basis. The system will manage content based on all the schedules served by a device, no matter how that device is allocated across the system. Tight coupling of the system components using native protocols and techniques developed for real-time environments ensures that even large systems are fully responsive to the operator with no delays through deferred command messaging for either editing or control.

The automation should allow operators, in central or remote locations, to view and manage activity on every channel. In a true multichannel environment, the simultaneous operation of multiple channels is enabled. The flexibility to use sources as required across different channels with no artificial restrictions is critical to providing a cost-effective and scalable infrastructure. Such an architecture allows additional channels to be added in the most cost-effective form, eliminating the need to introduce additional discrete operational areas — with the associated increased operational costs — as the system grows and diversifies.

For a system to be genuinely scalable, any expansion or upgrade must be able to be completed without any downtime. The system's redundant architecture should allow such work to be carried out one controller at a time without any total shutdown.

The automatic sharing of information across multiple applications is a prerequisite to implementation of large-scale environments and large numbers of playout channels. Integration is critical to the realization of economies of scale as broadcasters expand their operations. Unfortunately, such information transfers are affected today through proprietary application interfaces that are neither extensible beyond the applications they join nor are they adaptable. Yet the competitive exigencies require a unified application environment across business and technical operations.

As a response to this situation, SMPTE developed the SMPTE-2021 Broadcast Exchange Format (BXF), making integration easy and repeatable. Use of standards reduces the time, cost and disruption involved in process and infrastructure expansion. BXF makes it simple to achieve a high degree of integration between playout automation systems and other systems, such as scheduling and content distribution systems.

Unlike proprietary, closed interfaces of the past, BXF enables dynamic exchange of data. A good example of this is the ability for the scheduling system to update, delete, or add new events directly into the playout schedule. (See Figure 4 on page 39.) Using BXF, it is also possible to get new content into the delivery chain quicker, enabling timely reaction to changing schedule requirements.

The technology on which the standard is built includes the following:

  • Extensible Markup Language (XML), a general-purpose markup language, designed to facilitate the sharing of data across different information systems.
  • Built-in, informal extensible areas that provide the capability to add conventions necessary for specific implementations. Examples include European conventions of conditional access and DVB-related elements.
  • The ability to formally extend the standard at a later date.
  • SummaryTransport independence. The expectation is that the primary method of information exchange will be dynamic and message-based, using such tools as SOAP, a protocol for exchanging XML-based messages over computer networks. The standard, however, recognizes the conventions that exist in the broadcast environment and enables the use of file-based exchanges to accommodate in-place processes.

Revenue growth is the target of all media distribution operations. Growth is realized through the expansion of content delivery channels, so the playout automation system is critical to the reliable delivery of content across all channels. Systems exist that provide reliability — through features described above — and scale more than 1000 channels and as many as 64 operators in a multichannel architecture. These systems provide playout control of linear media and the management of file-based media.

Combining automation with other applications enhances productivity in workflows, creating operational efficiencies that allow broadcasters to expand channel counts and revenue sources while limiting expenses. Broadcast operations will realize greater efficiency and flexibility through such an integrated approach.

Combining applications into a single seamless application environment where content and application communications are shared provides greater scalability and commensurate economies of scale. This enables broadcast operations not only to grow, but also to grow more profitably.

Dan Hellman is product marketing manager at Harris Software Systems.