Facility monitoring

I spent an interesting afternoon in New York recently looking at the new central routing and switching system at ABC Television (Central Switching Center). One could spend a lot of time reviewing the details of how the facility created a unique architecture capable of operating at beyond 3GHz, but for this rumination on technology, I want to focus on the control and monitoring aspect of the system, which went live a few months ago.


Routing control system

One of the interesting aspects of the facility is that it incorporates technology from a number of vendors. The routing control system was chosen independently of the routers. There is more than one router because the facility chose to use a “path-finding” approach, with a central switch receiving signals from an input router, and feeding output routers for plant distribution and monitoring. The plant being replaced was approximately 3000 × 2500 in total. Considering the size of the plant, a single monolithic router would have to be enormous, probably in the range of 4000-squared.

The cost of a router goes up roughly in proportion to the product of inputs multiplied by outputs. For the sake of argument, let's say a 100 × 100 router costs $50,000. (It doesn't.) That amounts to about $5 per port. If that is a valid measure, a 4000 × 4000 router might logically cost $80 million.

The hope for a simple relationship breaks down, however, for several reasons. First, power supplies, frames and control do not scale the same way, so the total number might be half that, or substantially less. The technology is straightforward. You ask for a signal that is on the input list, and it simply shows up on the output. That costs money and leaves a lot of crosspoints unused, which are logically not rational. (HD sources to SD destinations, SD sources to HD destinations and key signals to camera return feeds are obvious examples.) So, at any cost, which no doubt would be huge, a path-finding approach was more rational and would not put the entire plant at risk of a single point of failure.

But what is different with a path-finding approach is that commands to switch a signal through the entire routing “fabric” become increasingly complex as the number of routers increases. To get a signal, the system must find and allocate ports on the input router, the core router and the output router. The idea of a non-blocking path is, of course, no longer possible, and the sizing must be done carefully. In this case, the three routers in the new plant are from one manufacturer.

This sounds simpler, except those routers connect to additional routers in control rooms, graphics, transmission and other areas of the plant. Those routers are from a number of manufacturers, making the control not so transparent. Maps of all of the available signals have to be in every router to make such an approach work, which is of itself not a simple task in a large and complex facility.

Vendor independence

The most challenging part of this concept may be the “cross platform” vendor independence. ABC chose to have all vendors write to published interfaces that they all supported. That was not simple, but at least vendors did not have to open their proprietary hardware and software to competitors' scrutiny.

The results were astounding (to me at least). It proves that protecting intellectual property and still responding to legitimate technology needs do not have to be mutually exclusive. Maintaining a system in which multiple software products must interact may be the ultimate test of this interoperability exercise. Indeed, migrating versions of software in the system after deployment and turn on is a tricky business. Even that has been carefully considered, and the installed system has a great deal of fault tolerance in the control system despite multiple vendors dependent on each other.

Monitoring

I haven't mentioned monitoring yet, but that too is part of the plan for the ABC facility, as it should be in all modern facilities. It uses “probes” on the input signals, and output signals as well, to return thumbnails and first-level measurements of all signals as they enter and leave the plant.

Another unique aspect of the system is that the routers are interconnected entirely on fiber. Of course, fiber presents a bit of a challenge for monitoring, so at critical points in the system, signals hit optical hybrids, which split the signal into a couple of outputs passively. A monitoring point is pulled from the hybrid and left on a jackfield for test purposes only. The active devices in the system are all monitored by SNMP, routers, audio embedders and shufflers, and probes. The operators use a sophisticated monitoring center to view reports from the monitoring system on close-up displays controlled through a KVM matrix, and on a large multi-image monitor wall.

The control room is used to check-in feeds from sports and news locations, and the equipment the operators have allows measurement and adjustment to ensure the highest quality. In an effort to simplify the operator's interface, ABC chose to have the waveform rasterizer controlled by the same unified control system that controls the router, proc amps and other hardware. In a very real sense, the monitoring and control systems have been tightly coupled.

Setting a high bar

Not all installations need to be this complex, either now or in the future. Small broadcast stations do not need path-finding routers and fiber interconnects. However, this facility illustrates what is possible when vendors and customers look at real-world needs with a clean sheet of paper.

All facilities can benefit from reduced apparent complexity that is abstracted by the monitoring and control systems. It allows operators to see the essential information without having to have access to the full depth all the time. Such systems also allow monitoring and control to be done in a centralized location rather than running to the rack room to adjust things that can be easily remoted to a single operating position.

The approach ABC took is not entirely without precedent, but in terms of scale, it is one of a kind to my knowledge. It sets a very high bar for monitoring and control, without which the system could not have been abstracted to the level it has been. Operators could not easily understand and control a system with as much complexity as this without well-crafted software solutions to do their bidding and return information necessary for understanding the system status.

John Luff is a broadcast technology consultant.

Send questions and comments to: john.luff@penton.com