File Access, Modification and Transfer Within Omneon Active Storage Systems
Broadcasters and media companies are moving to file-based workflows for ingesting,
producing, storing, processing, delivering, and managing their media assets. While in the
past, it was enough for a video server to record a video clip and play it back; today’s digital
storage and processing systems must enable software applications to perform diverse
file modification and transfer functions. Client applications that automate processes or
manage, edit, or process media need simultaneous access to files stored across multiple
file systems. The Omneon MediaGrid and Spectrum systems work together to provide a
media-aware storage infrastructure to empower these workflows.
Software developers can rely on the support of industry-standard file protocols by Omneon
active storage systems for the access and transfer of files. Common Internet File System
(CIFS) and Apple File Protocol (AFP) are supported for applications making OS-level file
access and transfer calls. In addition, File Transfer Protocol (FTP) is supported for applications
using this industry standard for transfers.
Beyond these standard protocols, Omneon develops and publishes open application
programming interfaces (APIs) for when software applications need media-aware functionality
or wish to develop more tightly integrated solutions than these standard file protocols
are intended to support. At the media level, these APIs support QuickTime® and MXF
wrappers so media files can be actively transferred for edit and review purposes while
being ingested. Software applications can easily unwrap media files and transfer selected
segments for restore and conform operations or flatten complex edit sequences down to
a single file. With the support of these APIs, clips can also be rewrapped or metadata
modified. For instance, more than a dozen audio tracks can be associated to a single
video track and closed caption or subtitling data can be inserted or modified.
This paper will delineate the options a software developer has in accessing, transferring,
and modifying files in Omneon active storage systems.
MediaGrid Active Storage
The Omneon MediaGrid is an active storage system that provides high bandwidth file access over an IP (Ethernet) network to any networkconnected
device, such as media servers, archive devices, editing systems, and browse applications. The system provides scalable storage
capacity, I/O bandwidth, file availability, and CPU processing power for demanding media applications. Clients requiring use of CIFS, AFP, or
FTP can communicate with a component of the MediaGrid called the ContentBridge, while an optimized file access scheme is available for key
client operating systems via a MediaGrid file system driver (see MediaGrid file access and transfer alternatives).
Spectrum Media Server
The Omneon Spectrum media server is designed to ingest and playout a variety of file types consisting of most video compression formats
used by high-end video cameras, edit systems, and broadcast transmission operations. Video compression formats such as DV, DVCPRO
25, DVCPRO 50, MPEG-2 (i-Frame and Long GOP), IMX, HDV, DNxHD, XD-CAM, and XD-CAM HD (Q1 2007), can be recorded (encoded), stored,
and played out (decoded). Video data formats, DVB/ASI, SDI, and SDTI can also be stored. Audio can be stored in AIFF, WAV, AC3, or Dolby E
formats. These audio and video essences are wrapped in QuickTime or MXF. Access to these files (or “clips”), for the purposes of interrogation,
modification, transfer, or deletion, is provided through the standard file access protocols and Omneon APIs (see chart below).
QuickTime and MXF file wrappers provide slightly different methods of storing metadata, and the Omneon Media and Player APIs buffer the
application from these differences and allow easy parsing and modification of metadata within these files.
File Access Protocol Standards and Omneon APIs
Omneon systems support either the use of open application programming interfaces (APIs) or standard network file system and transfer protocols,
such as CIFS, AFP, and FTP to query file systems, to read and write, open and close, and transfer and copy files. The Omneon Media and
Player API provide media clip-level query, parsing, building, rewrapping, and transfer functionality. These APIs allow the most common media
operations to be performed efficiently and with little development time by application developers.
Below is a chart showing the support of standard protocols and Omneon APIs by the family of Omneon active storage systems.
File Transfer Protocol (FTP)
Both the Spectrum media server and the ContentBridge of the MediaGrid system feature FTP servers. These software servers can be called
with standard FTP calls and typically, for Spectrum, provide the fastest method of file transfer for non-growing files. An efficient approach used
by some application developers is to use a combination of the Omneon Player API (for discovery of files in a Spectrum file system) with FTP
calls (for the transfer of files). Below is an example of a simple FTP file transfer from a Spectrum server through the ContentBridge on the
In this scenario, the transfer is initiated by the application server, and the media files pass through the application server on their way to the
destination Omneon system.
The FTP server running on the Spectrum also supports proxy FTP whereby an application can initiate an FTP transfer between the Spectrum
and MediaGrid where the file data will transfer directly between the Spectrum and MediaGrid without passing through the initiating application
Tail-mode FTP refers to an FTP process that transfers files while they are still being written. Tail-mode FTP can be used for some self-contained
files where frame sizes are relatively constant, no indices are interleaved in the data, and the files are written left to right. Tail-mode FTP allows
the source FTP server to transfer growing files. It works by polling the file size as it is performing the transfer, and as long as the file continues
to grow, the transfer process will alternate between pausing and transferring until the file stops growing. Tail-mode FTP can work as a standard
FTP process as in Diagram 1 or as a proxy FTP as in Diagram 2 when transferring files from Spectrum to MediaGrid. Tail-mode FTP is not
supported for transfers when the source file system is the MediaGrid.
Omneon Media API
The two Omneon APIs for accessing, transferring, and modifying media files are the Media API and the Player API (sometimes referred to as the
Player Control API). The main difference between these two APIs is that the Media API is a standalone library that typically executes commands
on the application server where it resides. The Player API simply provides an interface for invoking functions that are executed by the Spectrum
The Media API is available as a Windows® dynamic linked library (DLL) and, more recently, as a static linked library for Linux® distributions. It
offers interfaces for C and for C++ development. The two are equivalent in functionality.
The Media API typically sits on the client device along with the Windows or Linux application it is supporting and utilizes the processing power
of that client device to perform three main functions:
1) Querying clip properties - The OmMediaQuery class provides functions to get the in/out points of a clip, start timecode, and user data.
2) Transferring clips - The OmMediaCopier class provides functions to transfer clips, possibly performing “on-the-fly” rewrapping.
3) Modifying clip properties - The OmMediaQuery class also provides functions to set the in/out points of a clip, start timecode, and user data.
A new call extension added in Q4 2006 provides OmMediaCopier functionality to be remotely executed by the ContentBridge. It provides a
means for the application server, acting as the host for the Media API, to perform only the control portion of the tasks while designating a proxy
host (the MediaGrid ContentBridge) to carry out the data-intensive copy tasks.
The following diagram illustrates how remote transfers are implemented, in this case assuming a Spectrum to MediaGrid transfer:
The application will start a transfer by passing the details of the new transfer job to the ContentBridge. The ContentBridge will perform the
transfer independently of the controlling application. Transfers will be handled by the Media API transfer agent, so they will be performed in a
coherent fashion, (see Active Transfers) allowing other processes to read and/or play back the destination clip while it is still being transferred.
For remote transfers to function correctly, the ContentBridge must be able to access the source and destination server file systems as
mounted network shares, for example via a CIFS mount or with the MediaGrid file system driver.
The Player API file modification functionality has proven to be so helpful for software application developers that all of this functionality is now
supported in the Media API. This allows the power of Windows® and Linux® clients to be brought to bear on commonly invoked tasks such as
metadata updates, changes in start and end of media, subtitle insertion, audio track tagging, etc.
Automation, media asset management, and media processing applications take advantage of this off-loading of file modification to perform
complex application tasks inside their application servers. Media files may be transferred out of the Spectrum to an application server, modified
in the application server with the Media API, and then the modified files are transferred back to the Spectrum or to the Omneon MediaGrid
with the support of the Media API.
It is often useful for media clips, stored as QuickTime or MXF wrapped files, to be transferred and made available for editing or review
processes while they are still being ingested. Omneon refers to these coherent transfers (such that the beginning of the media clip on the
destination file system can be opened and read while it is still being written) as active transfers. The Media API supports these types of file
transfer operations, which are essential for today’s fast-turnaround workflows.
Active transfers require that transfers occur with growing clips and that the destination clip, typically on the MediaGrid, is accessible while
growing. While tail-mode FTP (as described above) can support active transfers for a limited set of self-contained file formats, the universal tool
used within the Omneon active storage systems for active transfers across all file formats is the Omneon Media API.
For reference files and for variable-frame-size (Long-GOP) self-contained files, the Omneon Media API is the suggested mechanism for
performing active transfers. A raw copy mechanism such as FTP is not designed to support coherent transfers of growing reference files. The
Media API will increment the header file while simultaneously transferring appropriate amounts of the separate video and multiple audio track
files. The Media API can perform the active transfer either on the application server where the Media API resides or the data transfer can be
performed in a proxy mode whereby the Media API initiates the transfer process and directs a MediaGrid ContentBridge (see below) to act as
the data transfer mechanism between the source and destination file systems.
It is also useful for applications to be able to transfer or copy only specific portions of files when saving transfer time is crucial and only a few
seconds or minutes of an hour-long clip need to be transferred. Omneon refers to these operations as partial transfers. The OmMediaCopy
calls in the Media API provide a facility for applications to unwrap and transfer just the portion of the essence files that need to be modified.
The call will also create a new header file as necessary.
The Media API supports file flattening (also referred to as sub-clipping or edit conforming) whereby multiple portions of a clip or clips can be
combined or flattened into one new clip. Files can be rewrapped at this stage, for instance between reference QuickTime and self-contained
The Media API also provides a class of calls, omMediaCopier, that will rewrap media files between self contained QuickTime, reference Quick-
Time, MXF OP1a, and MXF OP1b.
Omneon Player (Control) API
The Omneon Player API is available as a Windows®dynamic linked library (DLL) and as a Linux®static library. This API abstracts RPC calls from
the Omneon Spectrum embedded operating system and hardware. The primary functionality of abstracting these calls to Windows and Linux
APIs is to allow automation applications to more closely control the realtime ingest and playout functions of the Spectrum hardware. It also
allows this control to occur over commonly deployed gigabit Ethernet infrastructures. A timeline capability within the Spectrum software allows
commands to be queued prior to execution thereby ensuring frame-accurate execution of record and play commands at run time.
The Player API also contains calls for the discovery and modification of files in an Omneon Spectrum file system. Since it is making RPC calls
directly to the Spectrum MediaDirector, files are modified in-place directly in the Spectrum system where the files reside.
With the Player API, user metadata can be written to and read from QuickTime and MXF-wrapped files residing on the Spectrum media server,
start and end of message can be modified, and closed caption or subtitling can be inserted or embedded into the clip. Complex reference
files, which point to portions of several or many longer essence files can be “flattened”; that is, a new reference or self-contained file can be
created that includes new contiguous essence files (audio and video) of the precise duration needed to play the reference .mov or MXF file.
The Player API also provides callbacks for notifying an application of the presence of a new file opened for write in the file system. This allows
applications to immediately invoke operations on new files without having to constantly poll the Spectrum file system that might hold tens of
thousands of files.
By combining Player API calls for discovery together with FTP or CIFS support available on the Spectrum, reference QuickTime and MXF-wrapped
clips consisting of multiple files can be easily transferred via TCP/IP over Ethernet between Spectrums or between Spectrum and MediaGrid
Omneon MediaGrid API
The MediaGrid API is designed to provide application access to the unique features of the MediaGrid file system. Unlike the Player and the
Media APIs, it does not operate at the media clip level but at the file and file system level. It allows applications to modify Access Control
Lists (for user permission settings) and replication factors (the storage method for providing data resiliency on the MediaGrid). It does provide
a callback for notification of the presence of a new file in the file system, but it is not recommended as a mechanism for clip transfers under
In version 2.0 (available in 2007) the MediaGrid API will include the library of calls that allow media processing applications access to the
processing power of the ContentServers of the MediaGrid.
Also in the future, with the introduction of class-based replication in which unique replication rules can be applied to different classes of user
data, the MediaGrid API will manage the distribution of files across multiple MediaGrid file systems.
MediaGrid File Access and Transfer Alternatives
Developers currently have two fundamental alternatives for accessing files in the MediaGrid file system. The preferred alternative and the one
that leverages the inherent bandwidth scalability of the MediaGrid system is a MediaGrid client file system driver (or simply, file system driver).
This is accomplished by installing a file system driver onto the application’s client device. The other alternative is to use the standard file
system and file transfer protocol interfaces available on the MediaGrid ContentBridge.
MediaGrid File System Driver
By installing MediaGrid client file system drivers on each client needing access to media data on the MediaGrid, clients have high-bandwidth
connectivity without the need for hardware filers, accelerators or gateways. This is the optimal alternative for accessing MediaGrid files and
can also be used for initiating file transfers to and from the MediaGrid when an application is designed to process or modify the files during the
MediaGrid file system drivers are available for the primary operating systems used by applications in broadcast and media facilities. They
provide network redirect functionality for the client operating system such that MediaGrid appears as a network drive to the client. The
following operating systems and distributions are supported with a MediaGrid client file system driver.
Installing a MediaGrid file system driver provides an optimized point-to-point file transfer mechanism from the client device to the MediaGrid
system. The application can communicate with the MediaGrid file system and simultaneously provide the functionality supported by the normal
file system calls of that client operating system.
The file system drivers for Windows, Mac, Linux, and Solaris sit directly on top of the client’s operating system and direct requests to and
from MediaGrid. Applications make a normal system call to the operating system, which then hands off the request to the MediaGrid file
system driver to execute the command (i.e., read or write to a network drive). When the MediaGrid file system driver is loaded on the client, it
becomes intertwined with the OS of that client, intercepting relevant calls to allow MediaGrid access by the client. The use of the file system
driver is completely transparent to the application (this is also the same for other file system protocols such as CIFS and AFP). In essence, it is
performing a similar network redirect function as CIFS and AFP do. When a client is using the MediaGrid file system driver, there is no need to
use another protocol. Neither CIFS, nor AFP is needed to read or write data in this scenario.
MediaGrid File System Driver for Spectrum
The MediaGrid file system driver for Spectrum allows for direct transfer of media files between Spectrum and MediaGrid, without the need for
the files to pass through a third device such as an application server or a ContentBridge. It works using the FTP server resident on the Spectrum
so it allows the use of standard FTP calls and performs proxy FTP transfers. In order to maintain compliance with standard FTP clients,
the files are transferred using proxy FTP. Proxy FTP is generally used to transfer files between two servers. The FTP client tells Server 1 to get
ready to accept a file. It then tells Server 2 to send a file to Server 1. The two servers open a data connection between each other, and the
file is transferred directly between the two servers without going through the client.
The Spectrum MediaGrid file system driver uses this same mechanism to transfer files between the Spectrum and MediaGrid file systems. In
this case, the two FTP sessions (both send and receive) are on the same Spectrum host (/omfs is used as the directory convention for pointing
at the MediaGrid). In other words, the destination pathname and the source pathname indicate that a file resides either on the Spectrum or
on the MediaGrid. The Spectrum FTP server detects this condition, and instead of performing a true proxy FTP transfer to itself it internally
transfers the file between the Spectrum and MediaGrid file systems. This allows files to be moved between the Spectrum and MediaGrid file
systems by using a standard FTP client issuing standard FTP commands.
The MediaGrid ContentBridge
The optional MediaGrid hardware component, known as the ContentBridge, functions similarly to a NAS head in a traditional NAS system. This
alternative is useful in several scenarios:
1. when client devices are running on an operating system not supported by a MediaGrid file system driver
2. when supporting many older PC clients with low-bandwidth requirements
3. when client access is required prior to an application being tested with the file system driver
4. when application software requires FTP or Media API remote extension calls
The ContentBridge runs the MediaGrid Linux file system driver (see below) on its “back-end” and supports CIFS, AFP, and FTP protocols on its
“front-end.” The ContentBridge also supports the transfer mechanism (remote extension calls) of the Omneon Media API, so applications
using the Media API for partial transfers, active transfers, file flattening, or rewrapping can utilize the processing on the ContentBridge to enable
Using the ContentBridge CIFS or AFP Interface
Using this first alternative, client devices can network-mount the MediaGrid system using CIFS or AFP. This method is the simplest file connectivity
to MediaGrid and provides applications with the support of well-known operating system functionality for access, notification, etc. However
for access to MediaGrid by many clients simultaneously, it is recommended that the file system driver alternative be used as this will alleviate
the need for multiple ContentBridge devices.
Using the ContentBridge FTP Interface
The ContentBridge can also be used where applications are already written to support FTP. In this scenario, the file is transferred from the
source file system through the application device to the MediaGrid Content Bridge using FTP over a standard TCP/IP connection. Both FTP and
Proxy FTP are supported on the MediaGrid ContentBridge, as well as on the Spectrum.
Proxy FTP can be used for transfers between Spectrum and MediaGrid and can be employed when the file data does not need to move through
an application device for processing. In this model, the application device initiates the transfer between Spectrum and the MediaGrid
ContentBridge and the source and destination devices establish a direct connection and transfer the file data directly between one another,
reporting back to the application device once the operation is complete.
Using the ContentBridge for Omneon Media API Remote Extension Calls
Another use of the ContentBridge is when active transfers are required. Files are transferred across a similar path to Proxy FTP (above) that is
under the control of the client application with the media data passing directly from the source file system through the MediaGrid
ContentBridge. Data is then passed directly to the MediaGrid ContentServers using the Linux file system driver over an optimized TCP/IP path.
The difference in this method is that the intelligence of the Omneon Media API supports both partial and active transfers, as well as rewrapping.
Supporting standard file protocols combined with developing and publishing mature APIs — supporting media-level file modification and sophisticated
transfers — is essential to empowering best-of-breed software applications. This two-pronged approach has set the Spectrum media
server apart from earlier generation video servers and allowed over 100 software companies to develop tight interfaces to it for demanding
For these and other software companies developing applications around the MediaGrid active storage system, this provides a choice between
flexibility, performance, and functionality. CIFS and AFP can be employed with minimal integration effort. Faster transfers can be performed
with FTP. Sophisticated media transfers and modifications are supported with the published Omneon Media API.
This multifaceted approach also allows developers to quickly integrate workflow elements with Spectrum and MediaGrid systems and over time
develop even tighter and more efficient processes to give their customers distinct operational advantages.