Broadcast systems-support engineers and maintenance personnel are critical to 24/7 operations in radio, television, cable and other media—including streaming services, live events, gaming and now even data centers or “AI-centric” operations—regardless of scale.
The practices in broadcast support systems are in most cases sufficient (today) to keep systems up and running, but that’s changing fast. Many facilities will be adding new technologies, including artificial intelligence, VR and such.
Today, most broadcast facilities are pretty well set in their place to keep operations going—especially those sufficiently embedded in networking or IP (ST 2110) systems. But few have a systematic agenda that is written and used to train staff—regularly—on network systems, and even fewer have a “look to the future” agenda to prepare existing staff for their future.
System and Control Management
Osmosis is not the way to be ready for next-generation systems, so here are some ideas and thoughts on how to be or get ready for the next wave of technological challenges—applicable to both AI and advanced new media.
Redundancy and failure resiliency are the rule, inferring and mandating dual chillers or dual cooling units, depending upon the architecture. Backup power is provided to all available HVAC units, usually as dual panels or “A&B” systems fed by at least the main utility service and a changeover generator (“genpac”) system on-site, fed with at least a 24 to 48 hour fuel supply and/or a split natural gas/diesel option.
For the equipment racks’ systems containment room, plan on at least a “hot aisle-cold aisle” layout and an alarm system dual-routed to a monitored operations center—both local and off-site.
In many leased spaces such as high-volume (or high-rise) office complexes, where environmental systems are interconnected—i.e., spaces where offices and tech centers share the same HVAC systems, same monitoring systems and same support and maintenance teams—economy is the name of the game.
These spaces have conditions where temperatures are “rolled back” after a certain time (e.g., midnight), lights are reduced automatically or systems are put into a mode that only reports a subset of system-wide changes, including power reductions or circuit failures. In an “exclusive office-only” workspace, this may be reasonable to save costs or reduce continuous support staff.
This would be an ill-advised practice in any computer environment functionally engineered to work autonomously and unattended in a 24/7 world. This example would certainly apply to an “AI-centric” services platform in much the same way that a broadcast technical center (equipment room or data center) or command-and-control center must run continuously.
In the same way that sharing human environmental services in a high-volume technical space is a no-no, surrendering the controls or their monitoring to a “conventional” office management practice is also ill-advised. Some landlords will try to keep control of the power, lighting and cooling or heating systems under their own services and management. In an “office-only” environment, the risk of widespread systems failures or temporary losses that remain unreported until “the next business day” (morning) could result in a significant loss of data, systems or services.
If this is how your current or “planned to be” technical office space operated, be aware you are placing your business in a high-risk condition. Unless you have an active independent “secondary monitoring system” (controlled by you, the operator) that can take immediate action from any significant operational change, you are in the wrong environment for an “AI-centric” data center, broadcast center or 24/7 services business.
Know (and Trust) Your Security
It is not uncommon for control systems to be proprietary and engineered to the point of locked-out equipment rack areas with limited to no access for your maintenance and support personnel. And even if you could gain access to the control systems—and even if your own staff were “factory trained” to support the systems—some portions of the software systems may be “locked out” to prevent only the OEM (factory-certified techs) to gain any access for any modifications—even if seemingly routine.
Management-wise, you should only hire support or operational technicians with control experience or operational familiarity. Know the designated partner or contractors who can make “live” (or even remote) adjustments to quickly return your systems to normal operation before the systems begin an automatic or unauthorized shutdown, bringing the entire data center offline.
Inspection logs, especially those that are preventive maintenance (PM) in nature, should be kept and routinely inspected for accuracy and anomalies. Understand thoroughly what failed, what caused the failure and how the system was fixed. This is especially true when any circuit breaker trips or any UPS hiccups or fails to transfer, or if voltages drop to alarm levels. Today, agent-based AI systems learn from such logs and help preidentify or recall previous situations as an aid to troubleshooting or supporting the system during failure modes.
Emergency Readiness
Keep your internal teams up to date on any changes or additions to the systems, including alarms or significant HVAC or equipment-load changes, and know what to do with the UPS or generator(s) feeding the facility. Know and identify who to contact at the utility company (plus make that information accessible) should power be lost or if you are forced to lower local load.
Document and review how service teams are dispatched, who they are and how to contact them.
Ask yourself: Are they manufacturer-certified or experienced? Are there appropriate spares and are they current? Do your teams know and understand the software systems, both operationally and service-wise?
Is there full asset history available? Is it digitized and kept up to date? What are the expected steps in your SLAs (service-level agreements), and how far can you take the troubleshooting before contacting the manufacturer-certified support or contract service people?
If your system fails at any level, are your teams “ready to respond?” Are there “uptime contracts” under a formal SLA (Fig. 1) which can take over immediately, or within the “uptime” control parameter, i.e. they must respond in X hours or X minutes?
Upper management needs to feel comfortable that your entire team is competent and able to support and/or respond to any situation. Ask yourself: Are there enough resources and are there backups in case of outside illnesses or unavailability for any reason?
Are there sufficient “checklists” or “preminders” in obvious places that aid personnel in getting started “right away”—those steps which guide you to resolving issues (obvious or otherwise)—and how or if to act according to a contract or an SLA (Fig. 2 shows the typical context of an SLA).
If you haven’t put things like these suggestions in place, start planning now! Bring in outside support to walk you through setting up a sufficient plan (an SLA) to keep you alive and have a “lifeboat” ready to react if needed.
