08.08.2012 08:38 AM
Why Video Storage Is Different--An Introduction for IT Professionals
Introduction
Video workflows are very different than the enterprise applications that IT organizations are accustomed to deploying and supporting. Large video files and demands for fast access and real-time performance result in requirements for highly scalable storage systems with enormous bandwidth, consistently low latency and the ability to effectively support highly specialized video applications.

Attempting to force-fit traditional enterprise storage into a video workflow introduces the risk that the IT organization will have dissatisfied users unable to work trouble-free with the tools of their trade. Further, IT may face support challenges resulting from attempting to deploy storage where it simply was not designed to go.

This paper is intended to introduce IT staff to the unique nature of video applications and workflows, and to illustrate the unique requirements of shared storage for video workflows.
 
The Strange World Of Video Workflows
In the enterprise world you may be accustomed to focusing on applications, such as CRM, ERP, and email, and core elements such as databases and virtualization technologies. Though unstructured content is gaining in importance, in the enterprise world data is often block-oriented, and the relevant performance measure is frequently transactions per second.

The world of video is different. In the video world the focus is on workflows, where various specialized video applications are utilized almost like stations of an assembly line to process and distribute video content. In such an environment, it is critical to quickly move large video files between different systems in the workflow. Thus the key performance metrics are around latency and bandwidth, not transactions per second.

Years ago, video workflows were based on analog media: moving images were captured on film or analog video tape, and clips were physically spliced together to create new film or video materials in final form (e.g., TV shows, movies, commercials, etc.). In recent years, film and video have gone digital, and media production workflows have increasingly become file-based. As with many IT applications, the video processing infrastructure may initially use islands of Direct Attached Storage (DAS), but with workflows requiring that video files be shared across different applications and by different users, the ideal approach is to have high-performance shared storage at the heart of file-based video workflows. Figure 1 shows a generic video infrastructure to support a file-based workflow, in this case for a broadcaster.

Fig. 1: Example Video Infrastructure For A File-Based Workflow


As shown in Figure 1, there may be many elements in a file-based video workflow, and the key applications may include:
Ingest, which refers to taking raw video content that may have been shot with a TV camera, and converting it into video files that can be digitally processed.
Editing, where video clips are cut, enhanced, sequenced, mixed with audio, and further processed into a segment of final video that someone would want to watch, like a TV show or movie. Key applications here are products like Avid MediaComposer, AutoDesk Smoke, Apple Final Cut Pro, and Adobe Creative Suite, all of which typically run on very powerful workstations.
Transcoding, where video files are transformed from one format to another for editing or viewing on different types of displays (e.g., TV, iPad, iPhone).
Playout, where video files are distributed over satellite, cable, or the Internet for viewing by the intended audience.
 
The unique storage requirements for video workflows
There are unique, challenging requirements for shared storage to support a video workflow:

Predictable, real-time performance
Many video productions must run at 24, 25 or 30 frames per second, and storage must be able to reliably support the consistent bandwidth required to deliver these frames to the requesting application without excessive latency. There’s no extra credit for running faster, but running slower so that frames are “dropped” is unacceptable.

High bandwidth
Moving around many large video files can generate enormous requirements for bandwidth. Storage systems for video workflows are often required to support gigabytes per second of bandwidth.

Media application support
Media applications are different, and the storage system that works well with Microsoft Exchange or SAP ERP may flounder with video workflows and ingest, video editing, transcoding, or playout applications.

Effectively support hundreds of terabytes of large files
General-purpose IT storage is often optimized for transaction processing performance with small files, in environments where storage capacity is measured in terabytes or perhaps tens of terabytes. In contrast, throughput-oriented video applications use large files and storage capacities measured in hundreds of terabytes or even multiple petabytes. This different type of workload can put unique strains on shared storage.

Continuous operations
If storage is down, the workflow grinds to a halt. In the best case this means that a lot of skilled people have to take a long break, but in the worst case it means the TV broadcast becomes white noise, the news program can’t go on, or the video that thousands or millions wanted to watch online is not available. Shared storage for video workflows is mission critical: if the data is unavailable, revenue is lost.

Easy scaling
The ongoing rollout of HD and now 3D content is just the latest in a continuing series of advancements in video technology. Though each advance provides a better viewing experience, each advance also results in yet another type of video that needs to be produced, with larger video files to process and store. As these continuing advancements are adopted, requirements for bandwidth are continually growing within a video workflow, such that it is important that the storage system’s bandwidth can easily be increased without disrupting users or applications.

Ability to edit growing files
Some workflows have very specific, and very critical, functionality requirements. For example, when broadcasting live events such as sporting events, the ability to create a highlights package while the event is still ongoing is critical, and this translates into a requirement for the storage system to be able to edit a file while it is still being written.

Table 1: In Contrast: Enterprise storage vs. video storage


Enterprise integration and other basics
In addition to the above unique requirements for video workflows, the storage system must also meet some of the basic requirements of enterprise storage: • Comply with enterprise requirements for security • Integrate with enterprise management frameworks • Easy to manage • Simple to deploy Cost-effective

Vendor expertise
The unique nature of video applications and workflows makes it particularly important to select not only a product that meets the technical requirements, but also to select a vendor who understands this unique environment, knows the application partners, and has demonstrated success across a variety of demanding video workflows.

Storage Options For Media Workflows

Direct Attached Storage (DAS)
DAS can provide good performance for individual application silos, and offers a low “price per gig” compared to networked storage. However, as in enterprise environments, DAS does not effectively scale, can be subject to downtime, and becomes complex to manage when data must be shared. Though DAS is sometimes used to support an individual application in a video workflow, it is not a practical choice for scalable shared storage or fluid workflows.

Fibre Channel (FC) SANs
FC SANs with an integrated SAN file system can provide the levels of performance required by video workflows, at least initially, and are a viable option as centralized storage in a video workflow. Unfortunately, the predominant file systems used with FC SANs suffer from performance degradation due to file system fragmentation; SANs can be problematic to scale; and often most importantly, FC SANs are complex and expensive to deploy and manage.

Traditional NAS systems
Traditional NAS systems consist of two controllers (or “NAS heads”) in a failover configuration. The controller hardware defines the performance of the system and usually becomes a bottleneck for bandwidth: once a controller is “maxed out,” increasing performance requires either a “forklift upgrade” or a manual re-distribution of the workload across multiple systems. This fundamental limitation in scalability often makes traditional NAS systems a poor fit for demanding video workflows.

Clustered NAS systems
To address the scalability limits of traditional NAS systems, vendors have developed clustered file systems which can effectively join many controllers together and present them as a single image. If additional performance or capacity is required, then additional controllers can simply be added to the configuration. The scalability of clustered NAS systems makes them a viable candidate for video workflows.

Like their predecessors, clustered NAS systems are accessed via standard NFS and CIFS protocols. These NAS protocols can present a performance limitation that is insufficient for some video workflows, and make it very difficult to ensure consistently low latency. In addition, many clustered NAS systems are plagued with an inability to maintain consistent performance as the file system fragments.

Omneon MediaGrid
Omneon MediaGrid, a shared storage system by Harmonic, builds on the clustered NAS approach with a system designed specifically to meet the demanding requirements of video workflows. Like clustered NAS systems, MediaGrid uses a clustered file system to scale performance while maintaining a single system image, and can be accessed with standard CIFS and NFS protocols. However, MediaGrid goes a step further by being able to deliver higher performance via a File System Driver, which provides power users with true direct parallel access to multiple storage nodes through multiple Ethernet connections.

MediaGrid:
• Is built specifically for the unique and demanding requirements of video workflows
• Delivers higher bandwidth and superior price/performance compared to clustered NAS systems
• Maintains consistent performance over time, with no degradation due to fragmentation
• Consistently maintains low latency, with a tuneable file system and client-side caching enabling near wire-speed performance
• Scales non-disruptively to tens of GB/s of bandwidth and petabytes of capacity
• Is based on pervasive and simple Ethernet topology, providing the performance of SAN with the simplicity of NAS
• Is built for high availability, with no downtime required for hardware failures, software upgrades, or maintenance activities
• Has been tested, optimized, and proven with dozens of the leading video applications
• Is offered by Harmonic Inc., a leading provider of video infrastructure solutions with extensive experience in many of the world’s most demanding video workflows

About Harmonic
Harmonic provides infrastructure enabling content and service providers to efficiently create, prepare, and deliver differentiated video services for television and new media platforms. For additional information see www.harmonicinc.com.



Comments
Post New Comment
If you are already a member, or would like to receive email alerts as new comments are
made, please login or register.

Enter the code shown above:

(Note: If you cannot read the numbers in the above
image, reload the page to generate a new one.)

1.
Posted by: Anonymous
Sat, 16-13-2012 02:16 AM Report Comment
These devices are one of the must-haves for the merdon home entertainment center or home networks. They can perform multiple tasks from backups to media streaming. If you have a network, and just want a common place to store files, this will do it. Need a streaming media server for your new TV or networked media player? Ditto. Automatic network backup? Check. I'm using it for storing photos and videos. Since digital photos are only virtual (no more negatives for safety!), backups are a must. I keep several backups, as photography is a hobby and a job. I also have media streaming devices hooked up to my TV's, and they can play video, audio and show photos from the web and attached network devices. This device shows up on the list of network devices, and it's a few clicks to navigate to the media desired. Streaming works perfectly. Everything you need is in the box. A short networking cable, power supply, the network drive and installation CD. Installation is easy and straight forward. Attach the power and network cables, sit down at your PC, drop in the install CD, and follow the steps. You can't just plug in and go you need to run the install CD. This is not just a drive it's also a mini file server. You need to set up permissions and user accounts. The first thing the installer did was check for an update. One was available, so I let it install it. This took about 25 minutes. This step can be skipped, but it's always a good idea to install updates. After that was completed I created a couple of user accounts quick and easy. You can create private storage accounts, public, and pick who has access to what accounts. So little Billy can't see his sister's stuff, or each employee can have a private and public storage area. Really handy and well executed. Also included is backup software PC, Mac and automatic network. I did not install the software. I've used it in the past, and prefer manual backups. Case: If you know WD products, you'll notice that they all have the same look. Their media streamer looks like a miniature version of this unit. It's identical to their external drives. The user interface is browser based, so is accessible to anyone on the network with admin access. I've been using it for a couple of weeks now, and have not encountered any issues. Only one caveat when the drive arrives, you have 900 gig of storage. It may be a one TB drive, but only 900 gig is available. Warnings: 1) When you setup the drive, make a note of the IP address. The instructions for accessing via the network in My Computer in the manual don't work in Windows 7. You need to use the browser and enter the IP address. The shortcut also works differently in different OS. Windows 7 double clicking on the icon opens the shared folder. On XP it opens the dashboard / control panel for the mybook. 2) The only interface is the network port. Don't expect to attach anything to the device except the network cable and power. Comments: The interface, although web based, is a little clunky. For the price, you can't go wrong great bang for the buck.
2.
Posted by: Anonymous
Tue, 17-12-2013 10:17 AM Report Comment
Very interesting article, but not a big fan of articles written without author bylines. Who the author is, and her background is important. Blair Finkelstein, IBM Tucson




Wednesday 11:59 PM
Peer Profile: Tomaž Lovsin, STN, Slovenia
“Will there be a shift from coax to fibre? Or a mixture between the two which will require hybrid solutions to be implemented?”


 
Featured Articles
Discover TV Technology