The broadcast industry is undergoing a monumental change to digital broadcasting that affects not only signal transmission, but content, production and processing as well. Digital production, for example, requires a different medium (hard drive vs. videotape), and different processing equipment (digital servers and workstations instead of tape-based editing equipment) than analog production. Servers and workstations are much more susceptible to problems with power and cooling than their analog counterparts, and they don’t recover as quickly when stressed.
Broadcast facilities depend on mission-critical design to keep them up and running around the clock. Planning with prudence allows all aspects of a facility, including the crucial master-control room (like Universal Television’s network origination center, shown here), to operate without interruption. Photo by Andy Washnik.
With analog, a loss of power causes equipment to stop and restart where it left off, making the length of the broadcast outage directly related to the length of the power outage. But, in the digital world, a loss of power in excess of 50 milliseconds requires a reboot and a return to the proper point in the program before continuing. The actual broadcast outage can be much longer than the power outage. Digital equipment is also much more susceptible to power surges and sags than analog equipment, and must be operated in more precise cooling environments.
Large data-processing centers have a history of experience with these specialized electrical and mechanical service problems, and they address them on a daily basis. Data-center design has seen substantial progress in the past 15 years toward true round-the-clock reliability, where there is no tolerance for any system outages. The broadcast industry can take advantage of this body of knowledge without the attendant risk and learning curve.
The application of well-established design concepts from these data centers can provide mission-critical broadcast operations with a level of service far in excess of present systems. While it is more costly to implement a mission-critical design, the added investment is but a fraction of the total costs of the broadcast equipment and facilities. If broadcast-equipment manufacturers design all their products for ultimate reliability, the weak link then becomes the systems that power and cool this equipment.
Figure 1. Redundancy has progressed from the simplest system design of N+0 to hardened approaches with “system plus system” (N+N) capabilities. Click here to see an enlarged diagram.
Mission-critical design
Before designing any system, you must determine the level of mission-critical design the application requires. The ultimate system will reduce the risks to an absolute minimum, but it may not be appropriate or cost effective. It would be more prudent to begin by developing a “project business objective” that outlines the project requirements from a business perspective. This forms the basis for analyzing three design criteria — redundancy, maintainability and fault tolerance — so that they meet specific project needs.
Redundancy
In a power-delivery system, where N equals the load, the number of elements (N) and their configuration defines the system’s redundancy. As Figure 1 shows, there are many variations of redundancy. Redundancy has progressed from the simplest system design of N+0 to hardened approaches with “system plus system” (N+N) capabilities. The N+0 design provides merely sufficient system capacity to meet the load requirement, whereas an N+1 system has one additional unit. Systems with N+1 and N+2 redundancy require a system controller, which becomes a single point of failure. If one or two uninterruptible power supplies (UPS) fail (N+1 and N+2, respectively), there may be sufficient capacity to meet the load. But if the system cabinet should fail, the redundancy is lost. This deficiency can be remedied by employing an N+N configuration. In this design, there are two, distinct, independent systems serving the load. And either system can serve the entire load, even if its sister fails or is offline for maintenance or upgrade. Also, there is no tie between the systems, so there is no common device that can fail. This is the preferred system design for mission-critical applications. Computer manufacturers have responded to this preferred system design by providing dual-power-cord equipment so that System “A” can power one cord and System “B” can power the other. As you begin to implement digital technology in the broadcast environment, you will see dual-power-cord equipment. If you don’t plan for this design capability, you will end up powering both power cords from the same source, thereby compromising a basic reliability feature of this type of equipment.
Figure 2. In this non-redundant air conditioning configuration, there is only one condenser water path. If any item should fail or need to be taken offline for maintenance, the broadcaster must shut down the entire air-conditioning system. Click here to see an enlarged diagram.
Maintainability
The overall design and configuration of a system directly affects the ability of its owner/operator to maintain and upgrade it. Typically, broadcasters configure their systems to minimize outages caused by equipment failures. But, if they don’t design the electrical power and cooling systems properly, they can’t perform maintenance without shutting down operations for perhaps six to 10 hours. This often delays or even eliminates regular maintenance, increasing the likelihood of catastrophic failure. Most broadcast environments are dynamic, and they require continuous upgrades to change or expand operations. These changes also may require disruptive utility-system alterations that take days to complete.
For example, Figure 2 depicts a condenser water system with a normal non-redundant design. In this design, there is only one condenser water path. If any item should fail or need to be taken offline for maintenance, the broadcaster must shut down the air-conditioning system. By contrast, a configuration-redundant equipment design provides alternate paths that allow broadcasters to perform maintenance without affecting the system’s overall operation. Figure 3 shows a system that has alternate flow paths for the condenser water, providing 7x24x365 maintainability.
Figure 3. In this redundant looped air conditioning configuration, alternate paths are provided to allow complete maintenance of the system and to provide alternate paths should a piece of equipment fail. Click here to see an enlarged diagram.
Maintainability also requires developing a system configured so that if a normal item should fail, a backup can perform the same task. Referring to the re-designed chilled-water system, the red Xs and green Os show how the system design can compensate if an equipment-isolating valve fails to close properly (X). In such an event, there are still alternate valves (Os) that can isolate the equipment.
Fault tolerance
Human error causes over 60 percent of outages, so broadcasters must design systems to ensure that a staff member’s inadvertent erroneous operation of equipment doesn’t adversely affect the mission-critical broadcast operation. Broadcasters must configure redundant equipment so that, if it is unintentionally shut down, the shutdown will not affect other equipment in the redundant system. This design concept requires broadcasters to evaluate as many different scenarios as possible with the intent of eliminating single-action items detrimental to the ongoing operation. For example, don’t power all of the air-conditioning equipment serving an equipment room from a single piece of electrical switchgear. If that switchgear were accidentally shut down, it would affect all air-conditioning equipment. Figure 4 shows a concept called “checkerboarding.” It can help eliminate the previously described problem by ensuring that the air-conditioning equipment is redundant and fed from several different sources.
Figure 4. Normal vs. checkerboard distribution. Checkerboarding can help ensure that the air-conditioning equipment is redundant and fed from several different sources. Click here to see an enlarged diagram.
Different designs for different systems
Once you’ve set the mission-critical design criteria, you can design the individual systems. You may need to develop different sets of the three mission-critical design criteria (redundancy, maintainability and fault tolerance) for different spaces, and apply the designs appropriately. Equipment rooms, control rooms, studios, transmitter and uplink equipment, and other critical areas may not all need the same level of mission-critical design sophistication, because they all have different impacts on the final broadcast content – and, therefore, the business finances.
Many times, the “mission-critical design” only focuses on the electrical power system because broadcasters perceive it as the most critical system. Yet, loss of air conditioning is just as detrimental to the on-air performance of the broadcast operation. In a mission-critical facility, it is important to apply appropriate design criteria to the following systems:
- Incoming electrical power
- Standby emergency power
- Uninterruptible power
- Chilled water and air-conditioning systems
- Normal power distribution
- Emergency power distribution
- Uninterruptible power distribution
- Broadcast and electrical grounding systems
- Fire protection
- Fire suppression
- Building monitoring and control systems
- Security
From the beginning
Conversion to digital technology requires that broadcasters install this new technology while the analog equipment is still operating. Some broadcast companies are designing completely new digital facilities to minimize any risk to the ongoing analog operations. Such companies can implement mission-critical design criteria easily as they design and construct these new facilities. When compared with trying to implement mission-critical design later, after the facility goes online, this approach can significantly reduce costs. It is important to note that, in many instances, it is impossible to upgrade a faulty design to incorporate mission-critical design criteria without a major outage in the existing broadcast operations.
Leo P. Soucy, Jr. is president of Facilities Engineering Associates and a specialist in mission-critical design.
