Managing storage in file-based workflows

A media company's decision to implement a tapeless, file-based workflow marks the start of significant work for the facility's engineering department. A failure to define this undertaking in concrete terms at the outset can make it difficult or impossible for engineering to implement such a system successfully. Any individual workflow incorporates many discrete processes, and the goal of an efficient file-based workflow can only be realized when all these processes have been optimized to the greatest extent possible. To realize a complete file-based workflow, the system designer must include the concepts of centralized storage, clip management and overall process management, considering the impact of each of these on the overall system design and complexity.

Infrastructure considerations

The first step in migrating any facility to a file-based workflow is to replace the physical transport infrastructure, and this can be a relatively straightforward process of replacing the original baseband (and therefore real-time) interconnection of processors with IT-based connectivity schemes. IT-based backbones can offer significantly higher aggregate bandwidth between media processors, and that increase in performance, coupled with transportation of media as compressed data, can lead to an improvement in throughput. But simply replacing an existing video-centric routing scheme with an IT-centric routing scheme in and of itself cannot be classed as creating a file-based workflow.

Problems arise if the individual media processors in the facility (logging systems, NLEs, graphics systems, QC systems, transcoders, audio processors, etc.) still operate as processing “islands,” each with its own media storage subsystem that holds the media while the processor acts upon it. Reliance on islands of storage makes it likely that, at any point in time, a broadcast facility is storing a number of copies of any single piece of media — probably in multiple formats. In this case, the broadcaster needs a way to track and manage that material in order to maximize the efficiency of the operation.

A classic example in which islands of storage present a problem might be when a particular processor, such as an NLE, requires 40GB of local storage to hold a piece of media that it needs, but only 30GB are available. Though another NLE in the facility may have spare capacity available, the first system can't use that storage, as it's dedicated to the second NLE, which may or may not be from the same manufacturer. The “island” model would suggest that the broadcaster solve this problem by buying more storage for the first NLE system. A far more cost-effective solution is to share storage across this nonhomogeneous edit environment and allocate it to systems as needed.

To realize a complete file-based workflow, the designer must include the concepts of centralized storage, clip management and overall process management as part of system design. The rest of this article will examine the impact of each of these on the overall system design and complexity.

Centralizing media storage

A centralized storage architecture can offer huge benefits to any media enterprise. As a single entity, a centralized storage system offers significantly reduced management requirements, as storage can be allocated to external devices as needed and modified as the broadcaster's needs change. Such storage systems also can simplify expansion of storage capacity, as any added storage becomes available to any external device rather than being dedicated to a single function. (See Figure 1 on page 44.) When designed with an open file system, the storage system can allow multiple nonhomogeneous systems to access stored material.

For the central storage model to be effective, system design must address a number of additional parameters. First, as a central resource, the storage system cannot simply be taken out of operation for a software update, storage expansion or similar maintenance. If this were to happen, the facility would grind to a halt. Storage systems are available that offer “hot” upgrade capabilities, but the system designer should take this real-world issue into consideration and exercise care when choosing a storage product; otherwise, a backup storage infrastructure must be deployed to keep the facility operating while the central storage is offline. This scenario also requires that source material and finished material be migrated to and from the central storage in a managed fashion before and after the downtime occurs.

Another consideration when adopting a centralized storage system is bandwidth scaling capability. As the demand for available storage increases, there is an associated increase in the number of clients who wish to access that storage. In addition, new business requirements often dictate that higher bit rate files be processed by the facility, which means that larger files are being stored on the system itself and that increased bandwidth to the client will be needed to guarantee acceptable operation. Thus, the storage system needs to be able to grow in available bandwidth in a managed manner, or the inevitable result is that the system disintegrates into a number of islands of storage, each providing capacity and bandwidth to a constrained number of clients.

If the storage will be used as a replacement to local storage on NLEs, then experience shows that one of the crucial parameters for the central storage is access latency. Operators are used to a certain level of service in terms of immediacy of content — as provided in the past by their local media storage — and their expectation of a new central storage scheme, regardless of its scale, will be that it consistently provide the same or better performance.

Even in a facility that employs central storage, some islands of storage will likely remain. Ingest and playout servers, for example, may continue to operate on their own storage systems for security purposes. Consequently, the broadcaster still will need a solution for managing and monitoring the multiple copies of content residing on multiple file systems, if only to ensure that a particular clip can be appropriately archived or all copies can be deleted from the system as required.

Managing stored material

A crucial factor in any true file-based workflow is management of the material on the storage systems. There are many levels of content management, and each level offers a balance between ease of implementation (cost) and complexity of solution.

The simplest level of management is file-based, and in this case the user, librarian or administrator simply opens a window into the file system in question and interacts directly with the files. While this is basically a “no cost” option, it has several drawbacks. First, the client can only interact with one file system at a time. Without consolidation for the purposes of management, the broadcaster cannot manage the same files on multiple file systems. Second, users must understand the storage system's folder structure if they want to locate the material in question. Finally, and probably most importantly, users must understand the file structure of a specific media item. Clips are often made up of separate wrapper, video, audio, time code and VBI files. (See Figure 2.) If users don't understand which files belong to particular clips, they run the risk of moving or deleting the wrong files and creating problems that cascade throughout the workflow. A system that relies on user skill is unlikely to bring about significant improvements in workflow efficiency.

A second approach is to manage the storage systems at the clip level. In this case, the management system takes on responsibility for associating all of the files that make up a clip, and the users' interaction occurs at the clip level. Much more intuitive for the user, this model avoids the possibility of human error, such as the wrong essence file being moved or deleted, and allows some process automation to be enacted. Clips can be automatically moved, transcoded, archived or deleted based on a set of business rules. Once certain criteria are achieved (such as approval of an edit by an edit supervisor or completion of a file-based QC stage), an action may be initiated on the clip as a whole, with an associated improvement in overall system throughput.

Finally, the management of stored material may be approached as management of specific “assets.” In this context, an asset would be a conglomerate of media, such as all of the copies of a particular clip, regardless of location, format or bit rate. Through this approach, storage management can be extended enterprise-wide. A single search query can turn up all copies of a particular clip, even though those copies may be scattered across multiple file systems in the facility. Implementing this model yields substantive process improvements as the use of a single search operation, rather than individual searches on each file system within the facility, reduces the overhead involved in even the most mundane of management tasks. When coupled with clip-based storage management, the true efficiencies of a file-based workflow can begin to be realized.


A well-thought-out storage infrastructure is fundamental to a successful file-based workflow. Only when all elements are optimized can real workflow benefits be attained by the broadcaster. Solutions exist within the marketplace, and it's up to designers to evaluate each according to key design parameters and determine which storage system is most appropriate for their specific needs.

Paul Turner is vice president, broadcast market development, for Omneon.