A Multichannel Automation System

Shout “automation��� in a crowded television station and some folks still picture R2-D2 switching master control as the last human operator futilely resists assimilation by the evil Borg. While that may be the dream of some anonymous corporate accountants, the reality of today’s broadcast automation is far more benign and beneficial.

Automate:a verb: to make routine; to systematize, program, preset, or computerize.

A well-designed automation system helps staff manage complexity (and maintain sanity), perform repetitive tasks across multiple channels—usually at the same time—and accomplish more with available resources. It’s a workforce multiplier, in other words.

A BRIEF BACKGROUND
You can blame automation as we know it on the videocassette.

No more threading tricky tape paths by hand; operators pushed plastic cases into machines and stepped back. Soon after the patent ink dried, inventors began designing robotic gizmos to do the “heavy lifting” of pulling cassettes from their storage bins and inserting them into players. Of course all that required control software...automation software!

At one time, the business of television broadcasting was pretty straightforward: one station equaled one channel and its content—commercials and programs—came in boxes. Operators read labels and could hold a film can, cassette or tape reel in their hands; media was real and tangible.

In the early 1990s content started disappearing into electronics racks replete with flashing lights and odd whirring sounds: the first generation video servers. They gradually replaced the reassuring “touch and feel” of a day’s schedule on tape with CRTs listing the video server’s content. And of course that required new software to feed content into the “black box” and manage play-to-air. Automation once again.

Early servers had a relatively small capacity by today’s standards—a day or two worth of commercials, in most cases. That needed only basic automation (with a small “a”) to play groups of spots back-to-back on cue. As systems and users became more sophisticated, time-of-day triggers, as well as tape transport, switcher and peripheral device controls were added. Automation (big “A”) systems could operate the server, as well as switch feeds, play long-form programs from tape and even permit the operator to grab a quick break.

Today, multichannel operations are becoming the norm. Digital television, duopolies, local marketing agreements (LMAs) and centralization—coupled with advancements in server technology—are transforming traditional master control rooms into hubs of content, channel and asset management for entire facilities. At the same time, the “S-22 Committee,” a SMPTE working group, is hard at work redefining workflow and responsibilities shared between station traffic, master control and automation. A cooperative effort involving dozens of automation and traffic companies, the S-22 team is developing next-generation standards for bi-directional data exchange and dynamic, realtime playlist management—essentials for success in a multichannel world.

AUTOMATION TODAY
Broadcast automation systems have some basic elements in common: ingest and content administration, playlist management, and device control over the server and on-air peripherals. What separates merely adequate automation from the enterprise-grade system are the ability to gracefully grow larger, its range of functionality and redundancy offerings and the manufacturer’s commitment to customer support.

Creating a shopping list for automation includes some obvious ingredients, e.g. how many channels, peripheral devices, media prep and air control stations, and so forth are needed now—and in the future (hint: it’s usually more than you think). Intangible qualities are more challenging: user-friendliness, redundancy, a sufficiently “bullet proof” architecture, the ability to handle complex event sequences, accessible (open) databases, options for expansion and protection against technology obsolescence.

In a multichannel environment, the graphical user interface (GUI) must be carefully designed to help the operator maintain situational awareness of a large number of program streams, especially when staffing is lean. Simply pasting more playlist windows onto a traditional automation interface makes individual program status that much harder to follow. The GUI should also maintain legibility whether viewed on an operator’s workstation screen or spread across a “glass cockpit” display.

Redundancy, a consideration in any system design, is a major factor as channel counts increase and potential revenue loss from a system failure is that much greater. There are several interrelated ways to achieve redundancy. The gold standard is an N+N configuration: one or more backup server outputs are assigned to each primary channel and play out simultaneously under a secondary automation controller. If the primary falters, the backup automatically takes charge until the problem is fixed. The server can be configured as either a mirrored pair or as shared storage accessed by multiple input/output units. Primary and backup automation should always address separate server output modules. This type of redundancy protects against either a primary server or automation failure.

N+1 configurations offer a more economical, middle ground solution by using just one (or perhaps a few) spare server outputs. When automation senses a playback decoder fault, it cues the affected element to play from the spare port, switches that signal to air and performs an automatic join-in-progress. This method may not protect against an automation failure, unless a secondary control workstation is employed.

No redundancy is also an option. In some cases, depending on the cost versus benefit in a particular market, the expense of being redundant could be greater than any loss of revenue in a “worst case” failure scenario.

At the opposite end of the scale are facilities that assume the worst and plan for disaster recovery at a remote location. Essential content is mirrored to the independent disaster recovery site. To implement successfully, the automation system must be capable of moving media files around according to predefined rules and of keeping track of where everything is stored.

An automation system’s underlying design also impacts redundancy. If critical system “eggs” are in just one or two hardware “baskets,” what happens if one fails? Better architectures spread the risk across multiple workstations—a distributed processing model—virtually eliminating single points of failure. The automation computers themselves can be a “weakest link,” if their hard drives and power supplies are not redundant. Along these lines, an automation solution that supports SNMP (simple network monitoring protocol) is preferable to one that does not. SNMP-enabled applications permit background “health checks” of key components and parameters; out-of-tolerance conditions trigger an alarm before any permanent damage is done.

Another key factor in evaluating an automation package’s design is its adherence to accepted industry standards. Proprietary systems—for example, a video server that’s only controllable by the vendor’s own software, or automation that operates just a handful of servers and peripherals—are a risky choice, regardless of initial cost. It is better to select a vendor that supports the broadest range of equipment and has the development resources to certify new devices as they come to market.

As noted earlier, a key benefit of automation is helping to manage complexity. Channels often make extensive use of graphic elements and DVE moves for on-air branding and promotion. Most operators would be severely challenged trying to manually control all the devices in a typical end-of-program “squeeze & tease.” In a multichannel world, several such breaks often occur at the same time. These are typically automated as “secondary” (or “offset”) events, programmed in conjunction with a primary element in the playlist, but may have to be entered manually or have limits on how many actions are permitted in a secondary event. Multichannel-aware solutions, such as Sundance Digital’s FlexEvents provide channel-specific macros with an unlimited number of actions, and can be imported into the playlist directly from traffic.

As new content distribution models such as streaming video to mobile devices emerge, stations are adapting to a multichannel workflow environment where “content is king.” To make content and asset management effective, the automation database—which contains the majority of information about the facility’s media files—should be open to asset management applications, as well as report generation tools used by the business office. An “open standard” database structure, such as Microsoft’s SQL Server, is preferable to a proprietary architecture.

Regardless of an automation system’s size and complexity (or lack of it), service and support are critical. Broadcasting is a 24/7 business and, as most of us learned the hard way, our friend Murphy was an optimist. Automation—software that controls the very heart of your operations—has a direct impact on your facility’s cash flow. While most automation companies offer 24/7 support, the timeliness (as well as time zone) and quality of that support can vary significantly from one vendor to another.

Multichannel broadcasting is coming to a control room near you.

Steve Krant is VP of Sales & Marketing for Sundance Digital.