The inevitable full migration to generic IT hardware for broadcast video production is trying very hard to happen. Like an army of robots marching toward us over the horizon, rackmount servers, Network Attached Storage, blade servers and large network switches will be arriving en masse to take over the world of video production and programme preparation, finally supplanting videotape forever. But they march over sand dunes, not solid ground, and the progress isn’t occurring as fast as many would like.
It won’t be a simple, and perhaps not even a smooth, transition. A lot of broadcast engineers have been struggling with the planning for the new file-based production platform. One of the main sources of anxiety is the unknown robustness of very large encoded AV files as they are pumped from server to server, and sometimes re-encoded through transcoding engines.
Another major source of uncertainty is the “fly-by-wire” aspect of computer files, requiring a complex MAM (Media Asset Management) software layer to define what can and can’t be done with the files, and what can and can’t be known about the files and their history and their associations. Some programme producers who are in the process of “MAMifying” their video assets, if I may be permitted to create a new term, are finding that the sheer complexity associated with metadata management can be overwhelming.
Harris’ Quic But perhaps the most likely “gotcha” that might cause some embarrassment further down the line is to do with the robustness and integrity of the media files themselves. Will they last as long as a videotape on a shelf? Videotapes have lived on shelves for decades, and can be depended upon to reproduce the recorded contents reliably. Digital videotape is a hard act to follow because it has extremely robust real-time error recovery technology built into the playback device.
The recorded data on a digital videotape includes generous amounts of redundancy to allow very robust protection against data dropouts and media deterioration. Even in the worst case when a data read error is beyond the correction ability of the Reed-Solomon Codes, an error concealment technique makes the dropout unnoticeable, even for large burst errors on the videotape. The playback data-processing circuitry in a videotape player understands what the numbers in the data stream are supposed to represent, and can then use intelligent interpolation to create substitute pixels in order to conceal the gaps left by uncorrectable data read errors.
But an asynchronous MPEG-encoded AV file, on an IT server RAID array or on a data tape, is nowhere near as safe as its synchronous cousins on videotape. And there’s a multitude of reasons why encoded AV files on IT storage are at greater risk.
‘BIT ROT’ AND MORE
The first risky aspect of encoded AV files is the fact that the file itself is invisible to humans. The data file itself has no visible material form whatsoever. You can’t pick a file up in your hands. If the file is copied from one volume of IT storage to another, data corruption can occur. If the file is never copied, and just sits there idle for some long period of time (months or years), data corruption can occur due to ‘bit rot’ — a well-known fact, also known in the IT world as “silent corruption.” The still photography world has suffered the problem of bit rot and silent corruption for many years already. If you do an Internet search on “JPEG file corruption,” you’ll see how big the problem has become.
If the table of contents or file system for the volume that holds the file becomes corrupted, the file, though itself not corrupted, can become “lost” for all intents and purposes. In comparison, the XDCAM optical disc format has a special design that allows recovery of uncorrupted files even though the table of contents for the volume may have been corrupted. Generic IT storage hardware is not as robust. Videotape requires no table of contents at all in the data stream. The synchronous AV data on a videotape can be read from any start point to any finish point, unlike asynchronous files in IT storage, which must be read from the very beginning of the file.
And perhaps the most troubling aspect of the risk associated with IT files is the simple fact that careless human activity can so easily cause major loss, confusion and chaos in a large library of files that live in generic IT storage.
So, what can we do to make big libraries of large encoded AV files easier to live with? The first requirement is to have the ability to test the file for its data integrity. By “data integrity” we mean not only that the data is the same today as it was originally written some time earlier (no corruption during copy, transmission and long term idleness), but also that the data represents the correct essence that is needed for the AV content.
Checking for data corruption is an activity that lies purely in the IT domain. But sadly, the NTFS file system used under the Windows operating system doesn’t provide much in the way of features for maintaining high data integrity. The ZFS file system is much better in that regard.
Checking that the data represents the correct essence for the AV content is an activity that belongs firmly in the world of broadcast engineering and operations. The IT world can only tell us that the data in the file copy is the same as the data in the original file. But judging whether the data is valid video and audio, and is of the correct type and format, is not the job of IT. Only the broadcast operator can make the judgment as to what type of video and audio is correct, and what format is required.
No human would want the job of opening every AV file one by one to determine if the essence is of the correct type and format and that it isn’t corrupted. It would be a soul-destroying job to manage millions of files manually. It would be like trying to build an index for the Internet by manually opening every Web page in the world and reading them one by one in order to then manually build a catalogue and index. It’s beyond what humans can or want to do. And so the search engine companies like Google and Yahoo! have automated “crawlers” — programs that automatically visit every Web page and extract the index key words, and keep that index up to date by revisiting every Web page periodically, forever.
Tektronix’ Cerify To address the problem of rich media files in a content owner’s or broadcaster’s library, we need a similar concept to the “crawler” used by the Internet search engines. In the photography world, there are utilities such as ImageVerifier, which can be left to run overnight. That programme will open every photographic image file in the designated IT storage volume or folder and read it from beginning to end in order to determine that the structure isn’t corrupted, and create a report of those files that didn’t pass the test. We have the same concept in the broadcast world, but the cost is much higher because the volume of data to be checked is so much larger and the number of parameters to be tested is so much greater.
Broadcasters and production houses are now using, or at minimum, conducting trials on, products such as the Tektronix Cerify, the Harris Quic, the Interra Systems Baton, the Sony Ellcami, the Rhozet QCS, the Emotion Systems EFF, the Venera Technologies Pulsar, and many others. File QC for rich media files is now such a big problem that there are many companies chasing the prize for solving that problem in the most reliable and cost-efficient manner.
The problem of broadcast file QC is magnified because an automated workflow in a MAM-controlled platform will continuously generate new copies or versions of a file. A particular case in point is the expanding use of transcoding engines like those from Telestream, Anystream, Rhozet, and Digital Rapids, which all regenerate the AV essence by newly encoding the video and audio at a different bit rate, and sometimes in a different encoding scheme. For example, an edit suite’s output file in DNxHD format at 220 Mbps could be re-encoded to MPEG-2 long GOP at 15 Mbps for use in the playout servers. And the same DNxHD source file might also be re-encoded to Windows Media Video (WMV) at 0.5 Mbps and QVGA resolution for use on a Web page. Each new encoding of a file presents a new risk of a data corruption of some kind in the new file.
In a large programme preparation facility, typical of multichannel pay-TV broadcasters, the number of new files created every day demands the use of transcoding server farms, and downstream of those will be file QC server farms. We are now talking about the possibility of dozens of power-hungry rackmount servers or blade servers running at very high CPU utilisation for the bulk of a 24-hour day. This is a heavy cost in terms of power consumption, heat generation and rack space. It isn’t a trivial cost. It can have a major impact on the system design and the cost of running the platform.
Where possible, some systems can take advantage of existing unutilised CPU power to do the transcoding and the file QC. The Omneon MediaGrid, for example, can use each storage node’s spare CPU power to do transcoding in faster than real time, and file QC as well. This fact presents the possibility of a major saving to the broadcaster.
Especially in this day and age of “green consciousness,” system designers need to come up with clever solutions in order to minimise the size of server farms and all the power and heat and space that they involve. Transcoding farms and file QC farms need to be as efficient as possible so as not to become the equivalent of the smelly person in the elevator. Minimising the need for transcoding and the need for file QC by smart workflow design is, therefore, the first and most important step. Then when the bare minimum amount of transcoding and file QC has been defined, the most efficient devices to execute those tasks should be chosen.
The goal is to have perfect files at every point along the workflow from system input to system output. A bad file should never reach the playout servers. Data integrity and file QC have become key subjects for broadcast engineers. And a file should never be totally lost to the user. Backup strategies are a must, and in a way that wasn’t always regarded as necessary in the videotape domain.
Within this author’s awareness, the best work related to this subject is currently being done by PrestoPRIME in Europe. I would highly recommend broadcast engineers to familiarise themselves with that work.
