Operators confront shifting menace to IT systems
WASHINGTON: As if broadcasters, networks, and cable operators didn't have enough to worry about, a spate of summer computer worms made clear that a lack of diligence in network security can have results ranging from inconvenient to catastrophic.
The attacks by the worms--variants of Zotob, Bozori, Rbot and many more--failed to disrupt the on-air systems of CNN and ABC. But they infected, by most estimations, computers at hundreds of companies and institutions, and spoke to the need for media operations big and small to install their systems in an attack-resistant architecture and stay the course in protecting the network from the invaders.
HIGHER STAKES
As broadcasters move to more integrated, networked environments, the stakes get higher. Fortunately, most systems integrators have built the crucial on-air systems on independent networks with tightly controlled access to the outside world, and equipment vendors continue to work with customers to keep systems patched and running.
"The key to keeping the system as secure as possible is to eliminate or limit severely the amount of connections that you allow into your automation network," said Rick Stora, director of broadcast operations for Dallas-based automation company Sundance Digital. "The IT aspect of the installation has to be properly engineered, the automation network needs to be on its own subnet and on a competent router, [and] any connections to our systems have to tunnel through the router with permission."
Networks should connect to the outside world only through firewalls and virtual private networks (VPNs) and--this is supremely important--no one should ever check outside e-mail from a computer powering critical applications. Once systems are installed, administrators need to keep up the protection of the underlying machines with firewalls and install service packs and security patches as they become available.
But the patches can take a lot of time to install and can even require rebooting a computer to take effect--thus taking it off-line--although Microsoft and others are improving that situation, say several computer security experts. And the patches can affect the performance of some software, meaning that the software vendors have to "qualify" the patches, making sure the software can continue to run.
"There's this sort of battle between users and vendors, because vendors should be qualifying the patches as quickly as possible," said Harlan Neugeboren, a TV Technology columnist, news technology consultant and CEO of The Workflow and Technology Group. Neugenboren also noted the mutual interest of vendors and users to stay on top of the issue. "As updates and things happen, any potential patch needs to be addressed."
"There are applications and systems which tend to rely on older OS versions or which incorporate or certify patches slower than their operators/users might like," said Scott Teissler, chief information and technology officer for Turner Broadcasting System, which includes CNN. "This limits broadcasters' ability to implement comprehensive protective measures as rapidly as they would like."
Sundance tests the patches itself and reports back to its customers, even providing a link on its Web site to the Microsoft Security Upgrade Web page when circumstances warrant, as they did for the Zotob worm.
Avid, with thousands of installed editing systems, also qualifies virus protection software and all Microsoft security updates.
"We engage customers as early as possible in the sales process and actively seek the involvement of IT staff in network planning," said Jim Frantzreb, Avid senior product marketing manager for Broadcast and Workgroups. "Over time, we have developed a system of proven best practices for networking including the setup of firewalls, network segmentation guidelines, qualified virus-protection software, and staying current with software upgrades and security patches."
"We design those networks so they are not accessible outside of the corporate network firewall," said John Delay, Harris Broadcast's director of Strategies for Networking Businesses. "In most cases we'll actually design the automation and management control system as its own independent network that really never touches the outside world.
"As customers have been migrating to central broadcast environments where they're actually sharing media across an external network, this has become a key aspect of re-engineering security procedures and policies within the architecture for the H-Class platform," he said, referring to Harris newest content delivery system.
Once installed, Harris goes to "exhaustive lengths" to protect systems against hackers, fully testing patches in either Harris' Sunnyvale, Calif., or Denver labs before authorizing customers to install the patches.
"We're pretty vigorous about this because of the size of the installed base," Delay said. "Make a mistake and you'll take someone's operation down."
LO-TECH SOLUTIONS
IT experts also urge broadcasters to keep wireless systems super-secure with 128-bit encryption. Change passwords often.
There are other, lower-tech recommendations from the experts also, with outside e-mail cited as the biggest culprit. "Are you an individual who at work likes to go out just anywhere on the Web, and start surfing and looking around?" said Shane Coursen, senior technology consultant for Moscow-based anti-virus protection company Kaspersky Labs. "Well, if you are, you are taking the chance of infecting your system at work, and of course if your system at work gets infected, chances are that other systems at that workplace are going to become infected."
Los Angeles PBS affiliate KLCS learned a security lesson about a week before Zotob, when RBot breached its system.
Alan Popkin, director of TV engineering and technical operations at KLCS, thinks someone accessed e-mail where he or she shouldn't have, despite strict policy against doing so.
Popkin discovered some latency on some of the tapeless operations' 40 or so servers, worked backwards to find the problem, and took about 10 computers offline.
"We got lucky and caught it early," said Popkin, adding that the problem remained invisible to viewers. "If we hadn't caught it, it would have crippled the system."
It wasn't the video servers themselves that were affected as they operate on a Linux system, he said, but the virus can remain resident on them and infect the computers that control them. In the end, it took Popkin about two-and-a-half days to fix the problem.
HISTORIC WORM
Computer attacks first hit public consciousness in the early 1990s as hackers, many in their teens, wrote and released evil code as a sort of digital graffiti. By the late 1990s, many of the hackers had grown up and then got fired when the dot-coms busted, prompting a wave of malicious attacks, many from within companies.
Targeted attacks, with ideological or personal motivations, altered or shut down Web sites and caused similar destruction. In 1999, the Melissa virus used e-mails to multiply and spread and set the stage for a new generation of pests. In 2004, the Sasser worm found its way through unpatched Windows 2000 and Windows XP machines.
Malicious code today can take over computers, use them as nodes for shady activity or as "Zombies" for spam distribution, or introduce "keyloggers" to track a user's every keystroke and potentially steal sensitive information. Many of today's worms and viruses originate with those who have made careers on the Web's dark side--information theft and spam rings in China, Russia, the United States, Europe and elsewhere.
The Zotob saga began Aug. 9, when Microsoft announced a vulnerability and patch for Windows 2000 machines, of which there are countless units running in media businesses worldwide. On Aug. 23, Microsoft announced a similar vulnerability and patch for the newer Windows XP operating systems.
CNN initially judged its computer disaster important enough to break away from other programming and show live footage of office computers continually rebooting, and some say the high-profile reaction may have saved others from falling victim. An ABC spokesman said the problems there were patched within a few hours, and despite the use of electric typewriters to crank out copy, "World News Tonight" went on flawlessly. CNN said the worm affected about 1,000 computers--a small percentage of its total.
Zotob and its cousins may not have had the reach or overall impact on the Internet of some earlier worms. But it was historic, said Coursen, in that it got inside the walls of big networks and then wreaked havoc on those systems. Other worms slowed the entire Internet with significantly increased activity worldwide. Zotob had no such worldwide impact, but was a nightmare for its victims.
IT experts note that the cost of network security is rising in both time and money. "It's no longer a part-time job," said Dan Doggendorf, director of Enterprise Technology, Infrastructure and Security at Belo Corp. station group in Dallas. "And especially in the media business, security has historically been viewed as a luxury."
Within less than two weeks of the initial attacks, hackers in Morocco and Turkey were arrested. That was good news to network operators, but the speed of the worm, arising just days after Microsoft first announced the vulnerability, puts operators in a bind. System administrators can take all the precautions they like, "but all take a finite amount of time to deploy, and complex software mixes on some machines can inhibit deployment," said Teissler of Turner Broadcasting. "Regardless of this incident, so-called 'day zero' exploits remain possible and some businesses will need strategies to cope well with such events."
Said Popkin, of KLCS, "It's not going to be a simple, easy battle, because you have people out there who have nothing better to do than write viruses."
MOST READ
