In our current information/data age, the storage and protection of data is an extremely important part of any business venture. The degree of digital data being produced, globally, has long been outpacing the amount of storage available—irrespective of the medium onto which that data is stored.  

Cloud, obviously, is currently taking first place on the storage agenda given its flexibility and user expectations, however, there is a relatively new means for data storage that arrived several years ago and is now taking on some very different new perspectives.

We’re speaking of “DNA data storage”—yes “DNA” or more formally known as deoxyribonucleic acid—that tongue-twisting, nearly impossible to write term we learned back in high-school physics classes. That is, this storage concept is the enabling of molecular-level data storage into DNA molecules that leverages biotechnology advances in synthesizing, manipulating and sequencing DNA to develop archival storage. 

The next few paragraphs will take a very basic overview of the chemical make-up of elements in molecular science. You don’t necessarily need to be a biochemist to fully understand this section, but you can quickly see how this leads into the LOCO coding structure and error detection properties which are essential for DNA data storage.

The term “LOCO,” or in this case D-LOCO (for DNA-LOCO) means Lexicographically-Ordered Constrained Codes, which are “line codes that make it possible to mitigate interference, prevent short pulses, and will generate streams of bipolar signals with direct-current (DC) powered content” through the “employment of balancing.” These principles are found in magnetic-recording (MR) devices, in Flash devices, in optical recording, and in certain computer standards.

Ten Terabytes on a PinHead? 
Exploring the intersection of information technology and molecular biology has created a scientific means for the storage of over 10 terabytes of data in the space of a faint smear less than the space of 0.25 x 0.25 inches (0.0625 sq-inches). Comparatively, from American theoretical physicist Richard P. Feynman’s dissertation (December 1959), he theorized how to manipulate, manufacture, and control things on a micro-level (small) scale. 

From a coding perspective, this micro/nano technology practice dates back to work in 1948 whereby the density of data is increased through the use of such “constrained codes” which resulted in increased storage density in MR (i.e., magnetic recording). Such practices are still widely in use today which mitigate interference in today’s two-dimensional MR systems.  

Functional Artificial Objects
The science of this DNA storage remains somewhat experimental and is very much an ongoing research project of well recognized coding and by biochemistry experts throughout the world.

For example, using in-silico (i.e., experimentation performed by computer) and wet lab experiments, the Molecular Information Systems Lab (MISL) at the University of Washington (UW) in partnership with UW Computer Science, Electrical Engineering, and Microsoft Research, have brought together faculty, students and research scientists with expertise in computer architecture, programming languages, synthetic biology, and biochemistry to enable the use of DNA as a high-density, durable and easy-to-manipulate storage medium.  

Historically, the idea for DNA digital data storage began around 1959 when Feynman (see sidebar) outlined—at the annual meeting of the American Physical Society at the California Institute of Technology—the prospects of artificial objects of the microcosm and biological microcosms having similar or even more extensive capabilities in his paper “There’s Plenty of Room at the Bottom.” 

Another book worthy of further understanding is by Ed Regis (April 1996) called “Nano: The Emerging Science of Nanotechnology.” Nano tells the gripping story of how K. Eric Drexler and other scientists pioneered this emerging science. It explores what molecular nanotechnology could mean for our future as it was presented to scientists and congressional representatives on June 26, 1952.  These concepts were further recognized by future Vice President Al Gore, who presided over the presentations to the panel in 1996. 

Biochemical DNA & RNA
Functionality wise, DNA digital data storage is a process that encodes and decodes binary data to and from synthesized strands of DNA. Arguably, there is enormous potential resulting from its high storage density, but the practical use of DNA data storage is (currently) “severely limited” due to its high cost and very slow read and write times.   

In biochemistry, the depth of this topic is ginormous and well beyond the details presented in this article. However, the basics of what composes the DNA-elements include some of the following micro-biological concepts and principles:

A nucleoside is comprised of a nucleobase (which function as the fundamental units of the genetic code and five-carbon sugar (ribose or 2-deoxyribose). Nucleobases are nitrogen-containing biological compounds that form nucleosides, which, in turn, are components of nucleotides, with all of these monomers formulate (i.e., the constitute) the basic building blocks of nucleic acids.

A nucleotide is “a compound consisting of a nucleoside linked to a phosphate group.” The nucleotide is the molecular building block of nucleic acids, RNA and DNA, both of which are essential biomolecules within all life-forms on Earth. The four bases used in DNA are adenine (A), cytosine (C), guanine (G) and thymine (T).  In RNA, the base uracil (U) takes the place of thymine (T).

Ribonucleic acid (RNA) is a molecule that is present in the majority of living organisms and viruses. Like DNA, it too is made up of nucleotides—which are ribose sugars attached to nitrogenous bases and phosphate groups. This is a nucleic acid is found in all living cells that have structural similarities to deoxyribonucleic acid (DNA).  

In Fig. 1 the makeup of RNA (left) and DNA (middle) present in most living organisms on our planet and referenced to those 4-ary (i.e., a group or set of four [quad/quadary] data elements—in this case the “nucleobases”), are used in the coding of DNA/RNA for storage.  The CG-content is depicted in the smaller diagram (far right).

From the theoretic information perspective, strands of DNA serve as a storage medium for 4-ary data over the alphabet {A, T, G, C}. The “alphabet” referenced here aligns with the four components of the DNA nucleobases mentioned in the biochemical descriptions above and as shown in the diagram of Fig. 1.  

Cold Data
DNA is part of the next-generation technology that can support storing mass “cold-data” (i.e., “archival”) or information which does not require regular or continuous access. Pools of synthetic DNA are being proposed as a potential medium aimed at archival (long term) storage purposes. Through the use of coding and data processing, errors are prevented during the biochemical processing proceeding the development of the DNA strands.  

For long term storage, all of the data sequences must contain limited runs of identical symbols and a balanced ratio (percentage) of A-to-T (Adenine to Thymine) and G-to-C (Guanine to Cytosine) nucleotides. These compositions are referred to as “constrained codes,” which are a class of nonlinear codes that by proper processing, eliminate a chosen set of “forbidden patterns” from the codeword sets.  

 Promises and Encumbrances
DNA data storage promises formidable information density, long-term durability, and ease of replicability. However, information in this intriguing storage technology might also become corrupted. Experiments have revealed that DNA sequences with long homopolymers and/or with lower guanine-cytosine (i.e., “GC-content,” see inset of Fig. 1) are notably more subject to errors upon or when moved into DNA-storage. 

Guanine-cytosine content is the percentage of nitrogenous bases in DNA or RNA molecules which are either guanine (G) or cytosine (C). A higher GC-content level indicates a higher melting temperature. GC-content should be in the 30%-80% range, with 50-55% being ideal; and the GC content influences the evolution of proteins because of energy cost. A brief summary comparison of RNA vs. DNA are shown in Fig 2 (noting the CG-content in blue).

One probably never realized the complexities in storage beyond how magnetic areal density or how the fluctuation of magnetic energy relates to the pickup head of a spinning magnetic (hard) disk drive; but storage density will always need to increase if we’re going to sustain this ever-growing thirst for digital data. 

When or if DNA data storage becomes prominent is still a vision into the future, but it is being promoted by companies including American companies Illumina, Microsoft, Iridia, Twist Bioscience, Catalog and Thermo Fisher Scientific. According to Markets and Markets, the DNA data storage market is projected to grow from $76 million in 2024 to $3.3 billion—growing at a CAGR of 87.7%—by 2030. 

In recent years media organizations have been preparing for changes in how media is produced, which inevitably came to fruition once the impacts of Covid were realized. Prior to the pandemic, remote production was quite stable for those larger venue-like productions (concerts, sports, entertainers), however day-to-day operations (news, weather, local shows) had to be re-envisioned on a much broader scale once production crews became isolated.

The media and entertainment industry saw a huge impact as the industry nearly shut down overnight. Once organizations were able to survey the availability of their own resources, the extension of those resources quickly expanded, supported by software-enabled internet-based interconnects and flexible remote applications that were portable and adjustable.  

‘Remote Mode’
When the pandemic was “officially” over, the entities that enabled these new routine remote operations quickly understood that they could continue in the “remote-mode” fashion and sought to keep that new operational model functional. This obviously pleased the majority of those who had now resolved to work from home and could easily and effectively operate in that model with only marginal impact on homebase facilities. 

Certainly, exceptions popped up and adjustments had to be made—but the overall design and modeling for remote production was established. The foundation for many live and non-live productions were now setting like wet concrete.

Other things besides connectivity and applications have been changed or augmented to allow remote or at-home production to be sustained. The most obvious is the cloud—but cloud facilities had to expand in similar fashion to allow home-based central equipment rooms of the broadcast facility to be more easily utilized. 

By that I mean, accessibility, bandwidth, networking and fast, easily accessible storage all had to be reshaped in order to support live and non-live production requirements.

One of those areas, storage growth, has not slowed down in the least. Predictions say that by 2026 more than 220 exabytes of data (equal to 220 quintillion (1018) bytes) will have been created and stored somewhere in various ecosystems, according to research and a report by Coughlin Associates.  

The Varieties of Cloud
Reports have indicated that server and storage spending are leading indicators of the global economy. As for the future, companies are either terrified or optimistic of that pattern, or both at the same time, as predicted by NextPlatform in September 2022. IDC reported that total revenues in “shared cloud,” “dedicated cloud” and “all-cloud plus non-cloud” reached $39.9B; and overall increases from 2020 for the same period reached over $10B in growth. 

This only covers a single segment of those overall applications, and when compared with past sales—including service providers and everyone else (private users, non-shared services, etc.)—total revenue in 2020 was $130.9B and the predicted revenue in 2026 will reach $197.3B (references from IDC in 2022 as a CAGR of 10.9% from 2022 to 2026). 

With this growth, how will storage technology be able to keep up with the supply and demand expectations? What is changing to meet these goals? There is a steady change in storage technology platforms, which are necessary to meet the objectives expected. Some manufacturers are betting on the change to an all-flash storage device, whether for the local device or the cloud. One manufacturer recently reported that they expect capacity of their proprietary DFMs (direct flash modules) to increase by six-fold in a few years, reaching up to 300 TB capacities.

Advancements in 3D NAND (Fig. 1) areal density will look somewhat like how HDD (hard disk drives) grew a few short years ago. The physical capacities of such hard drives nearly reaching saturation as molecular densities drove magnetics to practically crashing and colliding upon themselves. Today, 24 TB and 48 TB DFM drives are shipping, dwarfing past other devices and showing that HDDs may no longer be the norm. 

How is this changing? Heretofore, 3D NAND devices used a stacking or laying principle with between 112 and 160 layers per IC. In the next few years (and less than five), fab vendors expect the number of active layers to increase to between 400 and 500 layers, yielding much higher-capacity 3D NAND ICs. 

Other Storage Segments
Applications for storage and its growth indicate that there are other storage segments which also need to be supported by M&E.

Long-term archiving will continue on various media form factors, with external HDD deriving around 18% of the archive storage space and 20% for local storage networks (NAS and SAN) with about an equal amount for private or public cloud at 21%. Digital magnetic tape (e.g., LTO or similar) remains the leader at 32% overall, according to Coughlin Associates’ research—but this may be shifting downward as cloud becomes more convenient and affordable. This prediction, again, is for long-term storage and for short-term storage the model changes even more dramatically. 

Both bandwidth and storage are required for a proper balance in content generation (i.e., production, capture, post and readying for transmission). The evolution of the network to 100 Gbps is almost routine now. Effectively, new installations striving towards an all-digital post-production environment should consider no less than 100 Gbps pipes to address the up-and-coming virtual environments that already require 20 to 50 Gbps connectivity/communications on a consistent basis.

Multipliers for content capture command more and faster storage solutions that can only be met by adding storage (local and/or cloud) and pipes that move data at rates approaching 50+ Gbps continuously. If you’re producing 4K (or 8K for long-term archiving) the network must be capable of handling 100 Gbps with the servers and storage able to receive (and return) such data rates. Today, for most storage systems in the M&E space, the bulk of those facilities continue to rely on HDD, however, SSDs are playing an increasingly important role through the use of NVMe SSD and NVMe over fabrics leveraging the “pooling” of storage.

Storage pools (Fig. 2) are principal constructs around storage virtualization. The “distribution” of a storage pool (a block of storage) may be connected in any number of means including Fibre Channel, iSCSI or direct attached (DAS) storage devices. The pool should not be “vendor-locking,” meaning that a pool should not be dependent upon one storage vendor’s product or proprietary solution—and instead allow flexibility to add or remove storage devices across the overall storage network solution. 

This, in turn maximizes the opportunity to scale or shift storage depending on the load or application; very important when exchanging files or resolution densities in a production environment.

One can see that the dynamics associated with these new and future production storage sets will require some careful planning to mitigate being trapped into a single source solution. This appears, on the surface, to be the target for future storage product providers, whether in the cloud or on-prem.

Protecting your data is a lot like your security practices. But legacy applications are no longer sufficient in this evolving digital transformation era. Some say now is the time to move on from old-school tools for data-management and look to new means for their data-protection solutions. This concept depends in part on how you and your organization view data as a valued commodity.

Storage systems and data products are a small part of the overall data-protection ecosystem. In recent times, some have claimed that so-called “legacy data-protection products” are little more than costly insurance. Up and coming new (and current) solutions can now help you address the most important aspects of data protection and help mitigate the complexity of keeping your data safe. Knowing the basics is imperative when establishing a path towards data protection.

Disruption in a New Light
The IT teams of today must look at their operations in a new light as nearly all industries—including entertainment—are being disrupted by entrants who are attracting customers away from their otherwise non-digital businesses. These new entrants can include “bandits” whose mission is to capture this data and exploit its value for themselves, including the illegal sale of that data or the ransoming of its usage by anyone, especially its owners.

The industry needs more help. It goes without saying that one of the new sets of challenges to IT-professionals includes finding sufficient workforce to support a growing set of digital requirements through the employment of its current (or newly hired) staff while continuing to protect the very assets that they manage or administer. Tools are available to aid in data protection, but it still requires proficiency and expertise in configuration and implementation of these new and usually advanced tool sets. The education and proficiency of these personnel are critical to securing the integrity of the data that they support.

Data is the lifeblood of any industry that uses digital technologies for its activities. It doesn’t matter what the business is or if it supports only a single entity or a myriad of other smaller segments within a global entity. Preserving your data is critical to survival, hence the IT industry must select from a multitude of products marketed to support and preserve this precious commodity.

Data protection is an essential function of enterprise IT management irrespective of the size or scale of the organization. As with any product you pay for, the intangible is almost untouchable until you need it, so let‘s first start by understanding what is meant by data protection.

For several years, storage solution specialists and vendors have recommended users include certain purpose-built commercial appliances in your storage architecture. Some of these legacy products were built to duplicate (clone, snapshot or other) and make available all your data sets in a quickly assessible, easily managed system that “auto-magically” would restore lost data from an independent archive platform with no losses or degradation in productivity. 

In actuality, these appliances or solutions were found to be a non-foolproof solution for maintaining data integrity, with the need, rationale and understanding of the requirements becoming a “missing link” in the overall solution set. As will be detailed later in this article—knowing about your data and the infrastructure it works within is a great first step to data assessment and protection methodologies.

Best Practices and Assessment
One of the suggested best practice approaches to ascertaining the validity or value of your data protection program is to do a Data Protection Impact Assessment (DPIA). This practice describes a process designed to “identify risks arising out of the processing of personal data and to minimize these risks as far and as early as possible.”

DPIAs are important tools for negating risk and for demonstrating compliance with things such as General Data Protections Regulation (GDPR) especially in those areas or regions with strict adherence to data integrity and preservation where GDPR is required by law. Those companies who must comply with GDPR are likely well aware of the data-protection requirements with many guidelines and practices published widespread. 

Automated processes include those principles, which are outlined in ISO 27001—the compliance document for international standard certification describing how to manage information security. ISO 27001 Certification demonstrates that your organization has invested in the people, processes and technology (e.g., tools and systems) to protect your organization’s data. ISO 27001 provides an independent, expert assessment of whether your data is sufficiently protected.

As an outline to assess if your organization is at least thinking in the right direction, the following elements are considered the “pillars of integrity” and can help you understand key issues in data-protection.

First, know where your data is stored or located, as well as how to retrieve it during compromising conditions. Second, understand the sensitivity of the data you have stored, the importance of the data to your business and the likely business impact should this data be unavailable or compromised to unauthorized parties (including the public). If the data should be modified or corrupted, know how to and what it will take to recover or restore that data.

Know How Your Data Flows
Third, owners should be knowledgeable in the prevention of unauthorized access and/or data transfer. This concept is aided best when you know and understand where, and how, data flows in your system and elsewhere (including the cloud). Unprotected environments are one of the biggest culprits to data loss and include not just the storage itself, but all those services that might have access to the data or involve interchanging that data, such as email, accounting systems, e-commerce solutions, backup and such.

Fourth, know and carefully manage the identities of all individuals who have access to the data. Determine if the right identities (both machines and persons) are permitted access and determine who should not have access to the data. Assess all third-party suppliers, partners and understand to what level they have access to the data including when, for how long, what kind of visibility they have to other sub-systems and similar security level concerns.

And finally, know what controls are being used to protect your data. Are those systems operating as designed and are they operating effectively? Are there stop gap methods that will immediately cease all access should any compromise be detected? Do you have your data backed up on premises or in the cloud or both? 

The Digital Experience
Today, there are two broad ecosystems that support data management—on-premises and in the cloud. The ongoing global “digital transformation” (see sidebar for a descriptive definition) is witnessing a monumental shift from on-premises workflows to those that include and/or are migrating to the cloud. Organizations today are highly focused on the digital experience and think they have a good understanding of their customers’ digital experiences.

IT decision makers (ITDMs) see an urgent shift towards focusing on consumer’s digital experiences. Such experiences are just like security, which now requires investments in technologies, applications, people and customer access to any and all information. If you follow best practices in security and policy, likely you’ll be amplifying the integrity of your data using similar practices. l

Sidebar:

Digital Transformation by Definition
Defining the Digital Transformation (aka “DX”) is complex and broad-reaching—best identified by example with no generally agreed upon fundamental definition. 

Essentially, DX involves how organizations and governments are changing their mode of operation in order to improve service delivery, be more efficient and effective in their designs, and achieve objectives such as increased transparency, interoperability and citizen satisfaction. 

During the pandemic, many organizations were not ready for the digital transformation. All forms of organizations have attempted to capitalize on modern technology architectures bent on adding flexibility and agility to expand their competitive advantages amid the disruptions and experiences of the COVID-19 pandemic. 

As some found out, any road to a digital transformation is seldom in a straight line. Results from expert interviews and reports indicate that “an amazing 40% of those organizations who commenced an all-out digital transformation initiative failed to achieve the desired business outcomes.” 

Configuring an end-to-end system storage solution for a new facility can be challenging. Systems once built upon the theory that “storage tiers” were key dividing elements—today, the challenges include managing data movement between such tiers and tailoring the storage types for best uses and applications.

Storage tier assignment was a model that separated “continuously-active” storage from “not-so-active” storage, i.e., fast vs. slow or bulk storage. Legacy storage tier designs continually pushed data amongst storage tiers based upon the associated activities known at the time of the design. As storage capacity requirements increased, more drives were “thrown” at the system.  Dissimilar drive capacities created differentials with performance measurement becoming unpredictive.  

As “cloud” and other sophisticated storage architectures came into play, a different science emerged. Previously dedicated “hierarchical architectures” changed functionality to leverage software solutions which manage needs, capacities, performance, and timing. By carefully modeling with best-of-breed data storage products, new solutions could be solved using complex data activity predictions and automation. No longer is it practical to use a single storage element or just the cloud, alone, to support modern media production activities.

Managing by Type to Fit the Need
Highly sophisticated structures in today’s media asset management solutions allow storage administrators to apply variants in storage sets across groups of individual topologies and workflows (Fig. 1). Software-driven approaches coupled with fast flash memory, and other backbones (NVMe, PCIe) now allow the administrator to select workflow segments and marry them into independent systems appropriately configured for specific workflows across the enterprise.

Add in a growing proportion of media technology which is moving to the cloud, and hybridized choices open up wider possibilities amongst on-premises and cloud workflows.  

Cloud supports ingest operations to funnel content from many geographic locations. Depending upon the need, placing accumulating data into a cloud platform offers a lot of extras, including long-term storage of content in its raw form plus the ability to consolidate content into a single virtualized allocation.

Knowing and defining the variety of steps to produce airable content is essential to determining if data is stored on-prem, in the cloud, or both. When and why to place which data in one or more locations vs. the efficiency or performance perspectives for specific or immediate workflows is what defines “managing by type and fitting the need.”

Differing Storage Structures and Elements
Organizations with varying requirements may need immediate, fast access to content at speeds incompatible with the capabilities of a “cloud-only” architecture; some workflows may not need the additional features, functionality or capabilities offered in a cloud solution. 

For media organizations, a workflow might simply push “raw” content straight into editorial production. When facing a breaking or live news requirement, they may need to push the new content straight to playout while other activities (including editorial) occur in parallel.

Often workflows must support multiple parallel processes, repetitive alteration, and documentary-like preparation for later specials or features.  In “live” production, raw content often needs at the very least a “technical preview” before being aired “live.” Producer approval or other reviews may also be required prior to pushing straight to air.  

Proxy generation, while simultaneously moving the data from an ingest cache to playout services, could gain efficiency from differing storage platforms. While some may only need a “tops-and-tails” segmentation, others may only require a mid-sequence removal or audio overtone.

These “fast vs. slow” example workflows demand the ability to quickly take in content, transform it to an “airable” format, and assemble it for purpose necessitates fast delivery process versus lengthier or more conventional tagging, shifting into editorial cache, or moving into near term storage for other production purposes.

Attention to Metadata
Tagging or metadata association for any number of workflows or production/legal mandates are not uncommon. Metadata collection covers activities from automated tagging to scene-change detection to detailed content analysis or evaluation with respect to people, location or purpose.  These variations gain additional benefits by their placement onto different types of storage depending upon the degree of effort required for each workflow.  

Such differentiation by activity helps reduce load balancing and may mitigate “bottlenecking,” which drags down other processes involving multiple read/writes or transfers between processing platforms. Other methodologies, such as Kubernetes and microservices structures, are leveraging AI principles to improve performance and speed up operations.

Depending upon the workflow, data may be repositioned to fast disk arrays based on the best system approach using the available targets. Storage that can materialize the information into a particular format, managed by a MAM, and sent to the cloud or straight to air ahead of editorial may also be sought. Thus, we recognize that “not all storage is created equal.” 

Monitor Proactively
High-availability storage systems need to be “watched” to ensure bottlenecks, over-provisioning, or other random events do not occur. Administrators who manage storage sets, manipulate the best data paths, and leverage file-based workflow activities proactively keep an eye on storage volumes, overall system bandwidth, and the management of those processes.  

Some less intense or non-immediate activities take more time to complete their operations and would best be allocated to hours when other commands needing higher performance are more immediately addressed. For example, if the storage volumes associated with editing command more reads and writes than simple reads for transfers—move those less important activities to overnight periods when less editing occurs. 

Administrators routinely must watch the logs to ensure that up-time, system loading, and security are handled proactively. 

Suggestions include scheduling upgrades at times when high-availability periods are unnecessary. Establish a hierarchy of notifications and reporting to ascertain when peak performance periods need full accessibility. High-availability work-periods, if monitored and reported proactively, can support most maintenance without downtime, and can occur during a flexible and convenient time based upon a routine schedule.

Extreme Hyper-scalability
Selecting the appropriate storage components requires understanding the entire media ecosystem. Workflows vary, so storage system administrators must engage components which are sufficient in scale to handle the average workflows yet still provide for those “hyper-activities” whereby the next level of availability can be enabled as necessary. Flexibility and capacity overhead for burst-up periods need consideration throughout the various workflows.

Speed and throughput are usually expected during “peak periods” but not necessarily at all times.  When a workflow pushes the envelope or approaches the edge of your storage infrastructure’s routine capabilities, your storage architecture should be expected to “spin up” to a higher level to address short-term needs. Changing when those activities occur (i.e., to “off hours”) may alleviate the potential for bottlenecks during peak periods.  

Some vendors can autonomously elevate their system performance by employing “capacity module” extensions. Additional support is usually configured using flash storage in an NVMe over PCIe architecture. These modules need not be deployed throughout all storage sub-systems; and are likely relegated only to your “highest-performance-level” activities. Plan for potential needs in massive scalability, also known as “scale out and scale up”—this extra horsepower may be necessary for one-offs or major events (e.g., Super Bowl, Final Four or breaking news). 

We’ve only touched the surface of modern intelligent storage management and its details. When considering an upgrade or new storage platform for a current or greenfield facility, be sure to take notice of the competitive and continual changes occurring in the storage-for-media sectors. You may be surprised how the many new on-prem storage additions are changing those capabilities which are essential to media productions. 

Why is a storage architecture an important component in the organization’s overall media composition and delivery platform? Simply stated, the storage architecture is the foundation that sets the prioritization of an organization’s data management, performance, metering and content protection strategy. The concept is relevant whether for transactional processes or unstructured media-centric content creation and delivery.

Storage management architectures are one of the more important areas that define both the ease of operations and the success of the operation from a delivery perspective. Additionally, unprotected or improperly structured architectures can end in disaster; and it is not unusual to find one or both of these elements in some of today’s content creation environments.

Storage technologies have steadily evolved over the past 30-plus years, as depicted in the accompanying evolution timeline in Fig. 1. Today, architectural styles for storage may be composed of solutions provided by storage service providers, storage component vendors and dozens of experienced (and some not so experienced) consultants or solutions providers. Given the system complexities, home-brewed storage at an enterprise level just isn’t very practical any longer. Selecting an appropriate style or foundation for your storage architecture becomes key to its performance and the continued success of your operations.

GETTING THAT CONTENT DELIVERED

As an example, in a broadcast television news organization the time to get a story “on the air” can make or break that program from an audience attention span and ratings perspective. Everything in a “breaking news” headline is based on who gets to see that story first. Coupled with a story’s promotion is the volume of delivery platforms that the story can get released in the fastest time. No longer does a news and information organization depend solely on its over-the-air or cable-channel distribution. News depends upon a multitude of delivery forms to get the message (and its advertising) to the user.

Delivery requires speed. Data generated from a single instance of a story must be conformed to many additional form factors—web delivery, social media, streaming services for phone and mobile devices, cloud storage (as applications) for other users or subscribers and, of course, the primary service from which it bears their name and logo of identification.

Each of the elements for this delivery will depend on a reliable and effective storage solution and its delivery methods. Any single failure can create a cascading effect that is unpredictable due to factors such as time, impact, audience attention and more.

How the organization picks its storage architecture depends highly upon how, when and where the content must be delivered—and who needs to get that content first. Such content delivery channels (paths) may be internal: that is, editing, production, post production, immediate on-air studio playout or next-time/repeat delivery playout. For external delivery that need may be for users who only see the content in a linear format; or for nonlinear services that repurpose and re-present that content; and many other varying avenues of consumption.

COMPETING PROCESSES

Copying, replicating, reformatting, transcoding, asset management delivery and other components are often utilized in both sequential and random processes. Such processes may require competing storage architectures along with a storage delivery network that can adapt to sudden changes, bursts in services and an unusually high level of continual stress that is seldom predictable.

Thus, picking the “lowest common denominator” for the storage solution isn’t the best, most efficient or most advantageous choice. Storage architects who configure these platforms need to understand the requirements and develop a solution (or set of solutions) that best services each of these needs in a flexible and adaptable environment.

As an overview, storage architecture styles may include “takeoffs” or examples of the following kinds or levels of storage systems.

TIGHTLY OR LOOSELY COUPLED

A loosely coupled system will not share memory amongst or between its nodes (Fig. 2 Green). In this system, data is likely distributed among many nodes, which may involve a large amount of inter-node communications during data writes. While simple to use and for distributed reads where data can reside in multiple places, such a system can be expensive when looking at the system cycles metric.

Transactional data (non-mediacentric data) can be impacted by things such as hidden write locations that are effectively low latency and brought on by the types of storage, such as NVRAM or SSDs. In this model, data may be kept in more than one location allowing multiple nodes to hold it and speed up access based upon paths that might be open, even if momentarily.

Conversely, in a tightly coupled architecture (Fig. 2 Blue), data is distributed between multiple nodes, often configured for parallel running and orchestrated by several high-availability controllers configured in a grid. Here, inter-nodal communications is necessary to keep the efficiency level high, latency low and processing running at peak capacity. Often such systems are engineered so that I/O paths are symmetric amongst all associated nodes.

In similar fashion to how enterprise class network switches function, storage system failures (drives, controllers, memory) are quickly identified, and alternative (prescribed) paths are implemented almost instantly. The effect goes unnoticed and operations continue without impact.

MULTI-TIERED AND CLUSTERED

In this model, applications (such as HTTP-based calls) will manage and make appropriate use of separate, almost layered tiers for specific delivery platforms. Web, application, database (asset management) and processing (transcoding or packaging) servers already have the storage access pathing embedded in their code-bases. Calls to central storage are routine, secure and redundant. Should any one tier be compromised, the other paths will be protected (i.e., are safe) with the aid of network security protocols, firewalls and even managed, physical switch segregation.

In the clustered environment, memory is not shared between nodes and data will “stay” behind a single compute node (Fig. 2 Red). I/O paths in a clustered architecture model may have varying layers. Some may employ an umbrella-like or federation model allowing the entire system to scale out as necessary. I/O is manipulated until the appropriate node reaches the data set needed for the particular task. Redirection code manages these operations with a potential drawback that induces potential latency while the right path awards the right connection for the selected data requirements.

DISTRIBUTED ARCHITECTURES

When a nontransactional data model is necessary, as in data in an editing, media asset management or a random processing chain (that is a nonlinear and nontranscoding operation)—a distributed storage architecture model across multiple nodes may be the choice. In this model, memory is not shared amongst nodes and the data is distributed across the nodes as in a fanout or multiparallel infrastructure. From a file system structure, such a distributed architecture may use objects and may operate as a non-POSIX (Portable Operating System Interface per IEEE Std 1003.1-1988) protocol.

Distributed architectures are less common but sometimes still employed by large enterprises, which enable petabytes of storage (Fig. 2 Green). Search engines and extremely complex asset management implementations with federated access on a global basis are candidates for the distributed architecture. In this situation, massively parallel processing models are implemented, resulting in speed and scalability—a perfect example of cloud-based resources when the onramp/offramp accessibility is unencumbered.

There are still more considerations for storage architectures that involve cost evaluations, compromises and scaling requirements—sometimes based on the phase of a particular project or the impacts of legacy and existing functioning components that still have a financial life expectation sitting on the books. Working with an expert in these areas is essential, especially at the enterprise level. Software applications, workflows and data structures all become key components in determining the full solution set. When the delivery of your assets depends upon speed, reliability and performance, one cannot afford to do things only half way.

Karl Paulsen is chief technology officer at Diversified and a frequent contributor to TV Technology in storage, IP and cloud technologies.

IT departments continue to look for ways to improve benchmarks for end-user devices throughout the enterprise, and in the media-domain that is no different. For media, especially editorial production, the need to create content that is funneled from various resources depends upon the workstation or server capabilities, but more importantly on the storage systems’ abilities to deliver the data faster and unencumbered.

A decade ago, the hot topics for maximizing storage capacity were centered on the thoughts and needs of “enterprise data storage.” There seemed to be little distinction between “storage for media” purposes and “storage for enterprise data,” despite the radical differences between media’s needs for high accessibility, large contiguous file sizes and uninterrupted delivery to editing workstations. Data was distinctively divided between “structured” as in transactional data and “unstructured-data” found in video and audio media (see Table 1). Approaches to managing these divisions varied depending upon the storage platform (NAS, DAS or SAN) and the volume of data to be managed on a usage level.

The stack of requirements for data storage management and efficiency included tiered storage, data migration tool sets and techniques such as data reduction and thin provisioning. Storage resource management was also emphasized during those times, ahead of the full acceptance and availability of solid state drives (SSDs) or NVMe (non-volatile memory express) devices (Fig. 1).

Initially, tiered storage was proclaimed as a first step for many organizations. Tiered storage was regulated by a process that took the least used data and moved it to a lesser accessible format such as linear tape. That model gradually evolved from on-prem physical tape libraries to deep archives in the cloud. Sometimes tape was retained for disaster recovery and legal reasons. Over time, and in some cases, the cloud would all but eliminate the on-prem tape medium.

PROACTIVE SERVICES

Storage resource management (SRM) uses a proactive approach that optimizes the speed and efficiency of the “available drive space” on a SAN (storage area network). For transactional data, the focus was on administrative procedures, which would automatically perform data backup, recovery and analysis. SAN solution sets often required a higher level of administrative maintenance, so SRM was furthered by implementing a combination of vendor APIs and a collection of the usual tools for systems management including SNMP, storage management initiative specifications (SMIS) and RESTful web services.

Utilization pattern data also emerged during this 2010 period. The tool set helped administrators determine how their storage systems managed input/output (I/O) requests. Through these tools, the systems could be improved by adjusting the way the drives were being utilized per the workflow groups to which they were assigned. Plug-ins were used to track real-time and trending patterns over any level of granularity (days, weeks or months) that the administrator wanted. For example, in a high-level rendering process for visual special effects, the demands on drive I/O would be continual but in editorial or color grading, the demands may be somewhat reduced.

LESSONS LEARNED

Many changes occurred in storage media between 2000–2010. The industry had started shifting SSD applications to include NVMe (2009) as the front end or cache portions of the drives arrays themselves, learning new lessons in space, speed and product fabrication. Even the migration from Tier 1 high-performance Fibre Channel (FC) deployment to less expensive Tier 2 SAS drives became more acceptable because improvements in the storage management tool sets allowed a more automated approach to implementation and administration. And once SSD arrived, the division between Tier 1 and Tier 2 almost blurred because I/O performance for SSD nearly equalized FC HDD performance, which the applications developers jumped on as a new and improved way of increasing performance of their own products.

Some of the lessons learned really leveraged rapid changes in drive performance alongside the acceleration of prolific content generation for streaming services—and for good reasons.

FAST FORWARD

A lot of things have changed over the course of the previous decade not only in storage device technologies, but also in performance, operations and administrative freedom as represented by shortened deployment time and user-sensitive interaction with storage management itself. Fig. 2 shows the relative transfer rate improvements from HDD through today’s NVMe storage sets.

Workflows are individual to the organization. Drive system providers are now focusing more on performance by adding simplicity to installation, configuration and administration. Features such as role-based authentication and single namespace architectures are taking the place of complex administrative activities that traditionally were the Achilles’ heel of the end users and which often kept systems administrators from going home at night or enjoying their weekends.

Software (not hardware) RAID is now supported by separating the controller profiles from the shares or pools of storage itself. Eliminating the dependency among the two systems enables yet another level of performance improvement. Desynchronization and data corruption is further prevented when using software solutions for RAID control itself.

Improvements such as hyperscalability, high availability (HA) and containerization support through application-specific interfaces are now easier and achievable, aided in part by taking the mystery out of the deployment factor. Through the addition of increased RAM and multiple CPU cores, more throughput and reduced latency are each achieved. Distributed file systems and clustering for media-centric storage implementation, typically reserved to high-profile compute intensive systems, is now expected.

Utilization of current optical fiber SFP connectivity—such as QSFP28 100G connections between the server engine node and the drive chassis itself—are pushing I/O (and IOPS) figures upward while being more readily adaptable to the user software systems for applications such as rendering, color grading and editing.

REMOVING RELUCTANCY

Nobody wants to or can afford to wait and with storage bottlenecks virtually eliminated using these newer and faster speedways—throughout the system—they are gaining acceptance across the enterprise.

Traditional IT departments who were reluctant to put different or new solutions in place, especially for M&E users, are changing their vision. Today, for example, deployments of new systems take only hours, not days. GUIs are easier to understand. Systems and their administration are more intuitive and quite different from what were previously used in older, established storage solutions.

Today’s users should expect a storage system to be as straightforward as their iPhone or Android mobile devices. Resiliency should not have to depend upon continual monitoring and tweaking of systems just to keep workflows fluid.

KNOWING THE RIGHT SOLUTION 

A key to understanding cost-to-performance models is in knowing the architecture of the storage system that you may need for your particular application. Stakeholders in the organization need to homogenize the users, administrators and technical support personnel in order to reach the right solution. A single drive set may no longer meet all the needs of the organization, however, that decision really depends upon the details of scale, diversity and the performance of the drive set and the solution provider’s unique capabilities for the tasks at hand.

Note: Portions of the technology discussions are courtesy of OpenDrives LLC of Culver City, Calif. 

Karl Paulsen is currently the chief technology officer for Diversified and a frequent contributor to TV Technology in storage, IP and cloud technologies. You can reach Karl at diversifiedus.com. 

Security guidance for ecosystems associated with data and storage systems had for many years focused principally on the protection of their associated systems, such as hardware, connections and processes used in backup or duplication. Other guiding objectives stayed fixed on the general support of evolving information security standards, per ISO/IEC 27000 directives and a collection of closely bound and integrated documents.

Standards have played important and strong roles in molding information technologies to become a harmonized set of criteria that has guided hardware, software and implementation. As a part of the entire ISO/IEC 27000 family of standards, a specific suite of documents is published by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) under the joint technical committees (JTC) in ISO/IEC JTC 1/SC 27 for IT Security techniques. The ISO/IEC established a series of continually developing documents, which grew out of industry’s rapid growth in IT, storage and networking.

The JTC was founded in 1990 when an earlier subcommittee (JTC 1/SC 20) moved outside of the field of security techniques associated with phrases such as: “secret-key techniques,” “public-key techniques” and “data encryption protocols.” These late 1990s techniques were later disbanded in favor of emerging techniques driven by new products, advanced needs and obsolete applications.

As is often the process in standards development, these root-1990 subcommittees, and beyond into modern day, have systematically altered various groups’ scopes and associated working groups’ efforts to meet current (at that particular time) demands for standardization. As such, numerous details, trials and tribulations often appear in more up-to-date applications as the technologies mature, emerge or obsolesce.

Data, in the form of assets, is of paramount importance to individuals, industry and enterprise. Many understand some of the founding principles (backup, for example); yet others have no foundational practice or reasoning as to why one approach is taken compared to another. This issue opens the topic of storage security as applied to using some of the ISO/IEC standards to support the safe keeping of media-related data and its practices.

CORE DATA PROTECTION

Like most network or storage system administrators, there are three core issues associated with data protection: privacy, information security and assurance, plus the storage (data) itself (Fig. 1). The goals are associated with protection centered on the notion that “all security goals and objectives be continuously maintained irrespective of any system complexity, variation in performance or the granularity of individual fidelity”—that is, without compromise.

Assuring all these goals are maintained is paramount to elevating the sustainability of the system, the integrity of the storage (i.e., its data) and the appropriate level of privacy, protection and information at all levels. When one realizes the broad areas such approaches might need to address, the details could be daunting. And this is precisely why the ISO/IEC groups assembled a series of documented guidelines and processes by which to follow.

However, the approaches can be confusing depending upon which versions or at what time your planning was first introduced. For example, in storage security guideline ISO/IEC 27040, the concept of “data protection” is not specifically addressed, despite a high degree of insight, which is presented in the document. Instead, this standard aims to establish security controls, which in turn helps elevate awareness of storage security through additional feature sets and best practices.

DATA PROTECTION CONTROLS

Since the starting point, file-based data workflows for media, audio/video and motion picture production depended upon and utilized simple storage management solutions with a modest degree of structure for its direct data protection. Early in that progression, users leveraged external storage solutions (e.g., removable drives) as a fundamental backup solution. Network storage was a “few and far between” alternative. These methods were sufficient, initially, but changed dramatically as storage volumes accelerated exponentially.

Likewise, only a modest number of users (and systems) had any serious concern for data protection—primarily because self-management and isolation was an easy, relatively straightforward process unencumbered by hackers, pirates and disruptors at that time. Hardware integrity posed more problems than content stealing prevention, due in part to proprietary codecs and self-regulated, insulated code support channels

Today, of course, we have an entirely different perspective. Exponential volumes of data have driven business continuity and disaster recovery requirements upward. Backup copies are now built to protect against data loss, one of a continually expanding set of data protection controls. Implementation guidance outlined in the ISO/IEC 27000 series (Table 1), prescribes organization requirements for information backup, including, but not limited to, software, systems and management policies.

Essential information beyond simply the data must be retained in facilities that can be run with the appropriate hardware and software tool sets to recover duplicate data sets should the original hardware be compromised. Recovery processes should be documented, itemized and their implementation procedures routinely tested should the main facilities be lost or destroyed.

Operational processes, such as how to monitor backup execution, must be reviewed against new software application versions whether locally hosted or in the cloud.

SECURING BACKUPS

At first the concept for securing backups, documented in 2013 under ISO/IEC 27002, may seem a bit archaic in the 2020 era of cloud-based protection, but they should not be forgotten or discarded by any means. In fact, the requirements for storage security, related to (secure) backups, are now clearly ingrained in the ISO/IEC documents.

Backup security is just one of the pillars for overall security on a broad scale. Today, it makes further sense to have a functioning system “in the cloud,” yet it does no good if the procedures are not consistent. Cloud vs. datacenter practices should often both be reviewed; have their routines updated and harmonized; and assure that all the code “tricks” created by developers remain in line with each other.

Extensions to storage security practices, which may now be ported to a cloud environment, could also be replicated in another “offsite” datacenter. Cloud may be for deep archive, with the data center being for rapid recovery—the choice is up to the business owner and administration.

TRUST AND VERIFY

Updated extensions include guidelines from a 2016 update of ISO/IEC 27004, which outlines “the effectiveness of measuring information security.” The 2016 update completely restructured the previous (original) document expanding it with a new purpose and putting it into current rules per the ISO/IEC Directives-Part 2.

Specific changes identified in the updates now lean toward media encryption, and operator authentication and authorization. They harmonize security practices with specifics identified for both backup systems and storage media. The concept of the “trusted” individual, component and/or system is now added, with the inclusion of a cleared and vetted or bonded individual.

Audit trail identification for backup processes and a clear path to verify the backup was actually performed are now required. Furthermore, a means to physically prove that restoration requirements are being met is necessary for proper certification. In effect, the backup is now trusted and the processes are now verified, per the standards.

FINER POINTS

ISO/IEC standards go much deeper and can be studied if you have access to or subscribe to the specific documents—most of which are not free. In part, data availability recommendations will include procedures for reliability, fault-tolerance and the requirements for performance of the data itself. Users must insure (and assure) against unauthorized access using technologies such as “data in motion” encryption, i.e., data that is in transit or in flight, is encrypted when the data is in the process of being moved (transported) between locations in either the facility, the computer and/or the network (Fig. 2). 

Like the practices and procedures found in well-known industrial efforts, such as in ISO 9000-certified facilities, should have similar concepts and approaches to those well entrenched in IT and storage solutions. Each of the ISO/IEC-subcommittee’s agendas helps to support the practices and provide sustainability across the industry with consistency. Appropriate follow through by all the storage disciplines should be expected. 

BLOCKCHAIN CHANGES THE COMPLEXION

Finally, we hear a lot of new uses and applications of blockchain, which is making data more secure, again. Transactional interchange of data that is untraceable, inaccessible and only known by the sender and the designated receiver—irrespective of the number of parties—is changing how content, contracts and transactions are being handled. The interchange of that data is so private and so secure that it is highly likely it will change the rules of storage and security forever. Stay tuned for upcoming topics on blockchain and how it will make the world a safer place for technologies.

Karl Paulsen is a SMPTE Fellow and frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at ivideoserver@gmail.com. 

Deciding, for the long run, where to keep an organization’s media-centric assets (i.e., original content, EDLs, finished masters, copies, versions, releases, etc.,) is reshaping the ways of storing or archiving data. Continued popularity in cloud-based solutions for ingest/content collection, playout from the cloud and processing in a virtual environment leads many to rethink “storage in the cloud.”

Yet cloud concerns still leave the door open to storage alternatives, tier-based migration, automated management and more.

What are the possible alternatives: Backup or archive? On-prem or in a co-lo? Private, hybrid or public cloud? Knowing the differences could create changes in how to approach archive management, regardless of the size, location or types of data libraries.

BACK IT UP OR ARCHIVE IT

Let’s look first at the differences in backup vs. archive.

Backup is a duplicate copy of data used to restore lost or corrupted data in the event of unexpected damage or catastrophic loss. By definition, all original data is retained even after a backup is created; original data is seldom deleted. Most backup just in case something happens to the original version, while for others, it is a routine process mandated either by policy or because they’ve previously suffered through a data disaster and pledge never to live through that again.

On a small scale, for a local workstation or laptop, practice suggests a nightly copy of the computer’s data be created to another storage medium, e.g., a NAS or portable 4-8 TB drive. Travel makes this difficult, so alternative online solutions prevail.

Businesses routinely backup their file servers (as unstructured data) and their databases (as structured data) as a precaution against a short-term issue with data on a local drive being corrupted. “Snap-shots” or “images” of an entire drive (OS, applications and data) are often suggested by administrators, software vendors, and portable hard drive manufacturers.

Incremental backups, whereby only new data or any which was changed since the last backup, are made due to the larger storage volumes and the time required for the full backup.

STORAGE AS A SERVICE

Archived data could be placed on a local NAS, transportable disk drive, an on-prem protected storage array partition, or linear data tape. Since an archive is about “putting the data on a shelf” (so to speak), the choices vary based on need.

Cloud archiving is about “storage as a service,” and is intended for long-term retention and preservation of data assets. Archives are where the data isn’t easily accessed and remains for a long uninterrupted time.

Archiving used to mean pushing data to a digital linear tape (DLT) drive and shipping those tapes to an “iron mountain-like” storage vault. In this model, recovering any needed data was a lengthy process involving retrieving the information from a vault, copying it to another tape, putting it onto a truck and returning the tape to the mothership where it was then copied back onto local storage, indexed against a database (MAM), and then made available on a NAS or SAN.

This method involves risks, including the loss or damage to tapes in transport, tapes which went bad or possibly had corrupted data onto the tape in the first place. Obsolescence of either the tape media or the actual drives meant that every few years a refresh of the data tapes was required, adding more risk in data corruption or other unknowns.

As technology moved onward, robotic tape libraries pushed the processes to creating two (tape) copies—one for the on-prem applications and one to place safely in a vault under a mountain somewhere. While this reduced some risks—such as placing duplicate copies in storage at diverse locations—it didn’t eliminate the “refresh cycle” and meant additional handling (shipping back tapes in the vault for updating). Refresh always added costs: tape library management, refresh cycle transport costs, tape updates including drives and physical media, plus the labor to perform those migrations and updates.

STORAGE OVERLOAD

Images keep getting larger. Formats above HD are now common, quality is improving, pushing “native” format editing (true 4K and 5K) upwards, added to dozens to hundreds more releases per program and keeping storage archive equations in continual check. As files get larger, the amount of physical media needed to store a master, such as protect copies and one or more archive copies of every file, is causing “storage overload.” Decisions on what to archive are balanced against costs and the unpredictable reality that the content may never need to be accessed again.

High-capacity, on-prem storage vaults can only grow so large—a hardware refresh on thousands of hard drives every couple years can be overwhelming from a cost and labor perspective. Object-based storage is solving some of those space or protection issues but having all your organization’s prime asset “eggs” in one basket is risky and not very smart business. So opens the door to cloud-based archiving.

MANY SHAPES AND SIZES

Fee-based products such as iCloud, Carbonite, Dropbox and such are good for some. These products, while cloud based, have varying schemes and work well for many users or businesses. Private iPhone users get Apple iCloud almost cost-free but with limited storage sizes. Other users prefer interfaces specifically for a drive or computer device with an unlimited storage or file count.

So why to pick one service over another? Is one a better long term solution versus another? Do we really want an archive or a readily accessible “copy” of the data in case of an HDD crash?

One common denominator to most commercial “backup/archive” services is they keep your data in “the cloud.” Data is generally accessible from any location with an internet connection and is replicated in at least three locations. However, getting your data back (from a less costly archive) has a number of cost-and non-cost-based perspectives. Recovering a few files or photos is relatively straightforward but getting “gigabytes” of files back is another question. So beware of what you sign up for and know what you’re paying for and why.

Beyond these common points is where the divisions of capabilities, accessibility, cost, serviceability, and reliability become key production indicators which in turn drive differing uses or applications.

ACCESS FROM THE CLOUD

Cloud-stored data costs have a direct relationship to the accessibility and retrieving of that data.

If you rarely need the data and can afford to wait a dozen hours or more for the recovery—then choose a “deep-storage” or “cold storage” solution. If you simply need more physical storage and intend regular daily or weekly access—then you select a “near term” (probably not an archive) storage solution. Options include On-Prem (limited storage, rapid accessibility) or off-prem in a Co-Located (Co-Lo) storage environment, referred to as a “private-cloud” or alternatively a “public” (or commercial) cloud provider such as Google, Azure, AWS, IBM or others.

Cloud services continue to grow in usage and popularity, yet, there remains a degree of confusion as to “which kind of cloud service” to deploy and for which “kinds of assets.” Many users prefer regular access to their “archived” material—this would be a wrong approach and is more costly (as much as 4:1) than putting their data into deep-or cold-storage vs. a short-term environment (“a temporary storage bucket”) that is easily accessible.

Fig. 1 shows a “hybrid” managed solution with local cache, multiple cloud storage providers, and local/on-prem primary archive serviced by an object-based storage “bucket” manager. The concept allows migration, protection, and even retention of existing storage subsystems.

CHOICES AND DECISIONS

Picking cloud service providers for your archive is no easy decision—comparisons in services and costs can be like selecting a gas or electricity provider. Plans change, sometimes often. Signing onto a deep archive becomes a long-term commitment due primarily to the cost of retrieving the data despite the initial ‘upload’ costs being much lower. If your workflows demand continual data migration, don’t pick a “deep” or “cold-storage” plan; look at another near-line solution. Be wary of long-term multi-year contracts—cloud vendors are very competitive, offering advantages that can adjust annually.

The amount of data you’re to store will continually grow. Be selective about the types of data stored, the duration you expect to keep that data, and choose wisely as to ‘what’ is really needed. Your organization’s policies may dictate “store everything,” so know what the legal implications are, if any.

Carefully look at the total cost of ownership (TCO) in the platform, then weigh the long-term vs. short-term model appropriately.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Aldermaston, UK, 19 November 2019 - GB Labs, innovators of powerful and intelligent storage solutions for media and entertainment workflows, today announced that the Office of Communications and Brand Management at Eastern Kentucky University, Richmond, KY, has purchased GB Labs’ FastNAS F-16 Nitro media storage system.

The office provides a wide range of services for the university, including communication strategies for projects that track with the university’s strategic direction and institutional priorities. Among those services are brand identity; public relations; web services; video production; social media; advertising; graphic design; news and promotions.

Eastern Kentucky University Production Manager/Producer Leonard Nave said, “Each member of our video production team serves in a variety of roles to provide services that support all of those areas, and that means everyone, to varying degrees, writes, shoots, and edits. If we’re not leading a team on one video production project, we’re serving as crew on another.”

Prior to its assessment of GB Labs’ FastNAS F-16 Nitro, each member of the video production team used multiple hard drives, daisy-chained to their respective desktops, all linked to a NAS designated for backup/long-term storage. As the previous NAS was so painfully slow, users could often be seen running up and down hallways with hard drives under their arms to share footage.

Nave said, “Our ‘sneaker-net’ file sharing system was not only inefficient, it often led to the creation of multiple copies of the same videos; and a library that was never fully or accurately catalogued.

“We urgently needed a video storage solution that would house all video assets in one place, enabling our entire team to simultaneously use that repository as both source and destination for all video editing needs. We also wanted local redundancy and support for off-site backup, plus extended hardware and software support. We found all of that, and more, in GB Labs FastNAS and its Nitro boost technology.”

Nitro is GB Labs’ intelligent hybrid disk technology that delivers performance close to that of pure SSD, but at considerably less cost.

A particular point of interest to the university was the new CORE.4 operating system at the heart of FastNAS. CORE.4 is a high-performance, custom OS specifically designed to serve media files with an additional intelligence layer that delivers stability and quality of service for every user. CORE.4 includes power-saving intelligence that uses the least number of disks whilst ensuring consistent, reliable performance. Its range of features and elegant user interface are also engineered to further ease the management of online workflows.

One of the major benefits, and a key selling point for Eastern Kentucky University related to CORE.4, is its compatibility with GB Labs’ award-winning “Mosaic” automatic asset organiser.

Mosaic integrates with AI tagging systems to automatically scour all inbuilt metadata to not only provide highly logical and swift asset organisation, but the intuitive ability to very quickly find and retrieve those assets. As an additional benefit, when applied to the management of student directories, it greatly simplified those processes as well.

Nave concluded, “The combination of GB Labs’ performance guarantee, FastNAS, and CORE.4 with Mosaic made the decision to go with GB Labs a ‘no brainer’. We can now concentrate on doing the work assigned to us instead of constantly worrying about the state of our video assets. It’s one of the best and most economic storage archive media systems from a single manufacturer.”


###

About GB Labs
GB Labs is the global leader in Intelligent Media Storage, creating a shared storage ecosystem for the media industry. By understanding real-world industry problems, cutting-edge technologies have been developed for the unique "CORE" software that fulfils end users’ needs. Regardless of where the production is being filmed, how big the team is or the size of budget, GB Labs can provide a solution to ensure deadlines are met and throughout the whole process, content is secure.

Find out more at: www.gblabs.com or call: EUROPE (+44) (0)118 455 5000 or USA (+1) 661 493 8480.


Company contact:
Matt Worth
GB Labs
Email: m.worth@gblabs.co.uk
Phone: +44 (0) 118 455 5000

Media Contact:
Kara Myhill
Manor Marketing
Email: kara@manormarketing.tv
Phone: +44 (0) 7899 977222

Scalability is generally attached to the concept whereby a system can expand from one particular “size” to another. Often the perception is that the top dimension is undefined—bringing to mind: “Just how large can this system expand to?” If you think about organizations (e.g., eBay, Amazon), there probably is no perception of the end point. Yet when applied to media asset management (MAM), the limits may be perceived by the number of records the system can handle, a capacity bounded by storage (not including the cloud), or by the effectiveness of the database to manage, search and retrieve the assets when needed and in a reasonable time period.

All this is quite ambiguous, to say the least. So, let’s put another descriptive term in front of scalability: “seamless.” Sometimes this becomes part of a marketing effect and sometimes the term is built in reality. In the case of the cloud the actuality probably can’t be identified regardless of the application—because “everything” is seamless in the cloud.

“Seamless scaling,” widely used to infer a system’s ability to expand to some level, is not necessarily a new term. Applicable models for the architecture are typically applied to networks, IP video streaming, CDNs, data storage and to MAM solutions. And most of these need to scale “seamlessly.”

When used in the “architecture” scenario, resources—and their usage—are, hopefully, scaled linearly against the load placed on the system by its users. For example, in compute operations, the load could be measured in the amount of user traffic; it could be set against input/output operations (IOPS); or may be related to data volume and more.

PRICE/PERFORMANCE CURVE

Resources must be balanced with performance. Here scalability is about resource usage associated with a single unit of work. Scalability, in this model, is about how resource usage and costs change when units of work grow in quantity or size. Scalability then becomes the “shape of the price-performance ratio curve,” as opposed to its value at any one point in the curve.

A price-to-performance ratio typically refers to a system’s (or product’s) ability to deliver performance for the price, capability or desire to pay. This falls into the proverbial “open checkbook” model, where cost becomes no object. However, there is a point where throwing in a boatload of cash ceases to deliver the performance desired in the time required. These factors impact the shape of that price-performance curve. See Fig. 1, where “time” can also be “performance.”

LINEARITY AND LATENCY

“Linear scaling” is sometimes depicted as a “straight line” model, its slope being determined by multiple factors—speed, throughput, latency, etc. In reality, systems utilizing technology simply won’t sustain a linear scaling model ad infinitum. Linear models will usually run at a given slope up to the point they fall over, dramatically change slope, become unstable or turn into a curve (Fig. 2).

Whether in a cloud or an on-prem datacenter, one variable impacting scalability is latency. Low-latency performance for workloads must be balanced with consistency. When latency dramatically changes, the net-net performance value is lowered (Fig. 3). Latency change is countered based upon design, system size, architecture, and the demand placed upon other services sharing the same resources.

One metric in cloud computing is its ability to deliver scalable access to a large pool of computational, storage and network resources, commonly known as infrastructure-as-a-service (IaaS). With functional workflows for storage, asset management, and databases moving more into the cloud, services leveraging machine learning and artificial intelligence make better owner/operator sense when they’re not built on premises.

Before the factors of price-to-performance are put into the model, a reality check about value proposition should occur and before the checkbook is opened or the bank account runs dry. It’s easy to let an unconstrained operational model go “cloud-wild” without implementing a set of checkpoints that help decide whether or not the practicality of going down this path is returning the value needed to reach the intended goal.

While the system (MAM, storage or network) may be able to scale seamlessly and linearly to a point without notice or consequence, users need to clearly put binders around the system that meets the goals in the time needed and without bankrupting the farm. Think again; perhaps the workflow storage solution is better kept at home.

“Super scaling” has been attached to hyper-converged compute platforms for at least a decade. Super scaling evolved from converged architectures as databases and analytics—regardless of their user applications—continued to demand real-time analysis for the delivery of its information and best performance.

Practical growth in these spaces previously relied heavily on massively parallel processors set into a clustered structure. The inhibiting force to this approach, in other than cloud, was the ability to access the storage at an I/O rate that matches the power of the combined processors, without latency or choking.

In terms of basic storage statistics and specifications, the rotational speed of the HDD, areal density of the bits, and the ability for data to be placed (written) or removed (read) from the storage material itself is not the whole story. Findings show other factors constraining scalability and performance for the storage system.

Solid-state drives (SSD) built on flash technologies, have changed the storage-dimension from what it was when only HDDs were available. Today, modern applications continue to push the envelope of storage I/O, capacities and processing. Seamless scaling now occurs in multiple dimensions aimed at supporting demand, change, data growth and adaptive user/workflows, all the time driving the question “cloud or datacenter/on-prem.” For the enterprise, direct attached SSDs (DAS) relay data to and from its servers, which strive to support an ever-expanding requirement set of transaction processing, data analytics and more.

DEMISE OF LEGACY DAS

DAS is successful because of the way it connects flash SSDs via PCIe, and continues to be a mainstream choice spanning over a dozen years. Unfortunately, the DAS+SSD scalability limit is nearly on the doorstep, to be relieved only by the comparatively recent technologies of non-volatile memory express (NVMe). Nonetheless, new methodologies still hold the constraints DAS had from infancy. Technically, if you only added a “shared-storage” model, then the value-proposition for NVMe became significantly diminished. Advancements in bus and I/O speeds changes that perspective, incorporating the principles of clustering and parallel processing for the storage environment.

Highly parallel, scale-out clustered applications require low-latency, high-performance shared storage capabilities. The latest change developed to address this weakness is that of the now standardized NVMe over Fabric (NVMe-oF). Fabrics, such in Fibre Channel or SANs, are software-defined resource topologies shared through interconnecting switches.

While all-flash arrays (AFA) changed storage models forever, its use of PCIe SSDs showed performance limitations when directly employed in application servers. Resources in this model became under-utilized to say the least—according to some, the net storage utilization averages between 30% and 40% and lower, in some cases.

Another issue in performance scaling is consistency. Some services, when run on a clustered server environment, can bring applications to a crawl. As services perform snapshots, cloning or other actions commanding CPU cycles, their functions take resources away from storage management activities—slowing data I/O and transfers.

Coupled with excessive data movement, complex operations and poor performance from large-footprint silicon devices—the negative impact on storage management cycles means that storage I/O becomes uncontrollable or variable to the point where errors develop, read/write cycles fluctuate or data becomes unavailable or worse, corrupted.

Recent improvements in scalable architectures, including storage, are supplementing advancements in physical storage footprints improving the capabilities to deliver data at the rates needed for workflows such as ultra-high definition (UHD), HDR/SDR and even the potential for uncompressed, high bitrate IP-flows (ST 2110) on servers and virtual machines.

In future storage articles, the depths of NVMe-oF will be expanded. In the meantime, if you’re looking at a storage refresh, investigate the (relatively discreet) manufacturers who are leveraging these new bus and data management technologies.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on storage and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

HDD technology has long dominated because of good performance, relatively high capacity, and low cost per TB with flash. As price differences between HDDs and SSDs decreased, the desire for higher capacity flash SSDs has become worth the discrepancy – especially as the TCO can sometimes be lower because of density and power.

That’s where NVMe, Non-Volatile Memory express, comes into play. Designed to unlock the performance of all types of non-volatile memory, from flash SSDs to the latest persistent memory technology, this primer will describe how NVMe enables greater performance in both standalone and networked implementations, and some of the key use cases that can benefit from this speed.

Download the full White Paper

In the many changes in media occurring during this age of digital transformation, production facilities are facing decisions about whether to go IP, while trying to determine the impact of producing content in UHD-TV1 (4K) and above.

In January, the 2019 Consumer Electronics Show featured evidence that 4K is here, stable and “readily” available—while looking squarely at 8K (UHD-TV2) as the next great change in television displays of the future. Of course, with 8K comes a need for creating and delivering that content—and, as the cycle continues, a quantum shift in how to efficiently manage the changes required to the infrastructure.

Obviously, the volume of bits needed to produce all this new content won’t really get any smaller. To add to it, the need to produce good content for 8K likely means it is shot in high dynamic range and at a minimum of 4K in resolution to realize the value of the larger screens, which will depend upon upscalers for quite some time.

Fig. 1 shows just how image sizes will relate to increases in bit volumes in order to meet the requirements to deliver high-quality video in the future.

MULTIDIRECTIONAL SCALABILITY

Throughout this past decade, applications associated with SaaS, AI, VR, social media and image resolution have demanded that systems scale in multiple dimensions. Peak demands continually change, forcing previously unseen growth in data sets and its management. New frameworks are needed to address these new data-intensive workloads in proportions that legacy solution sets cannot meet.

System resources are now being decoupled, allowing them to be scaled independently and turned into services versus how they were treated heretofore. Those who relied on the “shared storage” model are finding there is insufficient I/O (input-output) performance and excessive latency (i.e., throughput boundaries) to meet new demands, which goes for network interfaces and servers as well as storage.

Flash-based designs, now approaching more than 20 years since their introduction, are reaching a peak in terms of serial data I/O and transfers. No longer is it efficient to simply replace a spinning hard drive with an SSD and then tweak in performance on a legacy SAS or SATA interface.

Today, the latest up-and-coming technology is Non-Volatile Memory express (NVMe)—a scalable host controller interface coupled to a storage protocol that accelerates data transfers between client and/or enterprise systems that utilize high-speed PCIe (Peripheral Component Interconnect Express) based solid state drives (SSDs).

COMMANDS AND QUEUES

At least two primary factors affect SSD performance—commands and queues. A traditional SATA device will typically support up to 32 commands in a single queue; with the SAS device supporting up to 256 commands in a single queue. Here is briefly how these two constructs work:

Based upon the anticipated workload and system configuration, host software will create “queues” (positions or slots of availability)—up to the maximum supported by a controller. Often this is regulated by the core processor and is limited in number to avoid locking and to ensure the data structures are created per the cache of the core’s processor(s) without encumbrances.

A circular buffer, known as a Submission Queue (SQ) with a fixed slot size is used by the host to submit “commands” for execution by the controller. Each SQ entry is a command. A Completion Queue (CQ) is another circular buffer with a fixed slot size that is used to post the status for completed commands. The number of queues varies by the application—enterprise applications range from 16 to 128, with client queues only between 2 to 8. Block sizes for both are 4 KB (and above with NVMe protocols).

LANE CHANGES AND GIGATRANSFERS

Understanding the true meaning of the PCIe solution set can be complex, given the variety of interface opportunities and the interdependency of the I/O to and from the device being attached to the bus. Calculating PCIe bandwidth can be even more challenging, especially when giving rise to new terms such as Gigatransfers (GT/s) that are interchanged with rates and speeds, such as gigahertz (GHz). Rather than look at the explicit details of each “PCIe generation”—we’ll provide some introductory information about the evolution of PCIe in a simpler perspective.

Platforms utilizing PCIe connectivity have continued to rise, moving from Gen1 (24 lanes) to Gen2 (36 lanes with a doubling of bandwidth), to an I/O performance of 1GBps per lane in Gen3.

A lane is a data-transmission link, which consists of two pairs of wires—one pair for transmitting and one pair for receiving. Consumer PCIe slots can be 1, 4, 8 or 16 lanes. Packets of data move across the lane at a rate of 1 bit per cycle. The 1x link (one lane) carries 1 bit per cycle in each direction (hence two wires per direction times 2). A 2x link (two lanes) utilizes eight wires and transmits 2 bits at once (per cycle) in each direction. The numbers grow with successive PCIe generations.

PCIe Gen1.x and 2.x use 8b/10b encoding, which results in a 20% performance overhead. The encoding converts an 8-bit data set to a 10-bit character set, thus a per-lane 250 MBps bandwidth can only carry 200 MBps. When utilizing the gigatransfers parameter from the gigabytes bandwidth number, the numbers have a non-uniform change that is best presented in a table, which reflects the encoding, transfer and speeds for comparison (Fig. 2).

SSDs supporting Gen2 with eight lanes, deliver over 3 GBps; doubling that on Gen3 interfaces to 6 GBps for a single device. Encoding is now 128b/130b, resulting in only 1.5% overhead. Latency is also reduced and the ability to directly attach to the chipset or to a CPU was also revealed.

IT DOESN’T STOP THERE

AMD announced at the 2019 CES it would be the first to support PCIe 4.0 at either the enterprise or desktop (client) levels. Ironically, in May of this year, the PCIe 5.0 specification (providing four times more bandwidth than PCIe 3.0) was announced even before Gen4 (PCIe 4.0) was shipped.

Just how far will this go? Why does the development continue when, for example, Gen5 essentially has the bandwidth of a 100 GbE (Gigabit Ethernet) connection—equivalent of about 63 GB per second at 16x speed? Certainly, home users don’t need these values and even enterprise users might question this proposition—especially given the limits of the SSDs they would connect to and the costs of the switch gear—irrespective of the continually decreasing costs.

One evolving scenario for these exponential data rate growths is that of the “always-on” (vs. the “always-connected”) compute platform level, which is taking shape with the always-on model seeming to lead the race because of battery life expectations. For the media and entertainment industry, i.e., those routinely creating UHD/4K content, storage refreshes can now take less space, require less power and cooling, and increase performance multifold over previous storage solution sets.

EMERGING PROTOCOL

Current storage system advances are now based upon the latest NVMe protocol. Utilizing NVMe with multicore processors aids in removing bottlenecks that conventional interfaces alone experience. NVMe brings about highly scalable new capabilities for accessing storage media at high speeds, which in turn is enabling more growth for data-driven marketplaces including media, video production and post.

When you’re in the refresh mode or considering moving to UHD production, take a look at storage solutions incorporating NVMe, especially those who offer the repurposing of existing SSD, which you may already own.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Aldermaston, UK, 12 August 2019 - GB Labs, innovators of powerful and intelligent media storage solutions for the media and entertainment industries, has announced that North America’s National Hockey League (NHL) has purchased GB Labs’ MiniSPACE SSD 1RU storage system.

One of the drivers for the NHL’s move to GB Labs was the league’s need for high-performance storage coupled with ease of transport. Its existing storage system had proved to be unreliable, suffering from poor performance and inadequate portability.

GB Labs North American Channel Account Executive John Alaimo said, “A key component of what appealed to the NHL about SPACE SSD was its ability to provide simultaneous live ingest and direct connectivity for multiple editors, which opens up numerous opportunities for the multi-platform delivery of stylised content. Also, on top of the speed of the SPACE SSD range, it provides the performance, portability, and reliability they had previously lacked.”

Following discussions, the league opted for GB Labs’ MiniSPACE SSD 1RU to take advantage of its light weight, high-portability, fast SSD, and the reliability and performance of GB Labs’ acclaimed CORE.4 OS.

The NHL’s MiniSPACE SSD 1RU storage system directly supports eight editors; a data wrangler connected by a high-performance 10GbE in order to receive 1000MB/s I/O; and a 24-port switch for instances when extra editors are required.

The GB Labs system enables the NHL production team to easily transport its storage system and reliably edit entirely on location whenever required.

###

About GB Labs
GB Labs is the global leader in intelligent media storage, creating a shared storage ecosystem for the media industry. By understanding real-world industry problems, cutting-edge technologies have been developed for the unique "CORE" software that fulfils end users’ needs. Regardless of where the production is being filmed, how big the team is or the size of budget, GB Labs can provide a solution to ensure deadlines are met and throughout the whole process, content is secure.

Find out more at: www.gblabs.com or call: EUROPE (+44) (0)118 455 5000 or USA (+1) 661 493 8480.

GB Labs company contact:

Matt Worth
GB Labs
Email: m.worth@gblabs.co.uk
Phone: +44 (0) 118 455 5000

GB Labs Media Contact:
Kara Myhill
Manor Marketing
Email: kara@manormarketing.tv
Phone: +44 (0) 7899 977 222

Basingstoke, UK — October 25, 2018 - EditShare a technology leader in intelligent shared storage and media management solutions, today announced that The University of the of West England (UWE Bristol) has upgraded the existing infrastructure of its new Arts, Creative Industries and Education media programs facility to a modern, end-to-end, file-based workflow powered by EditShare Flow media asset management (MAM); a 256TB scale-out, high availability no single point of failure XStream EFS 450 shared storage solution for the Film/Animation department, and a 128TB single node XStream EFS 300 shared storage solution and GEEVs ingest and playout server for the university’s Broadcast Journalism program. The advanced, high-performance infrastructure features extensive automation and provides seamless integration with the university’s third-party non-linear editing and graphics systems as well as multi-platform distribution of student-created programs.

Dick Allen, UWE Bristol Faculty of Arts, Creative Industries & Education Technical Resource Manager, Digital Media & Broadcast, selected EditShare to create an immersive infrastructure that connected systems, content, students, and staff. Dick states the importance of having a Media Asset Management workflow, “Managing several programs and hundreds of broadcast and film students requires a detailed level of asset tracking as well as automation capabilities to have an efficient operation. Implementing a media asset management layer is key to optimizing the infrastructure, enhancing collaboration between students and staff, and giving our students a real-world media experience using modern tools.”

At the heart of the new UWE Bristol facility is the EditShare XStream EFS 450 and 300 scale-out storage platforms and Flow media asset management facilitating content access and sharing across the university’s production studio, two large photography studios, animation stop-motion and CGI studios, sound recording and Foley studios, colour grading studios and editing suites which are all connected to the EditShare XStream EFS. The university’s news broadcast studio, which also leverages the EditShare XStream EFS and Flow solutions, is equipped with GEEVS for multi-channel playout to TV and Internet channels. The EditShare powered infrastructure offers an integrated workflow that is student-optimized, providing hands-on experience that mimics real-world workflows that are tightly integrated with the program’s expansive course content.

“EditShare creates an immersive learning environment by connecting systems, content, students and staff,” explains Tara Montford, Managing Director, EditShare. “Flow, XStream EFS and Geevs form the core solution that assists students, staff and research teams to find, reuse and store media assets. It enables collaboration of teams with seamless integration of media systems and embedded tools to retrieve and archive media from a central location. With metadata tracking and asset indexing, staff and students can easily locate and re-use content, increasing the value of asset use.”

UWE Bristol leverages real-world content creation applications including Adobe Premiere Pro CC, Avid Media Composer, ProTools and Blackmagic DaVinci Resolve to give students an experience-based curriculum with an intelligent media management layer that brings efficiency to the university’s expansive program. Flow organizes the massive amount of content created and exchanged, acting as a control layer across the EditShare XStream EFS scale-out, high availability shared storage platforms, which are designed to support 4K, 8K, UHD and beyond with near infinite scalability on a no single point of failure architecture. Flow automates media ingest, transcoding, content searches, online to offline movement of media to ensure students are using the correct format and that their content is copied to the right media space.

In addition to providing the core infrastructure housing all UWE Bristol media program assets, the XStream EFS 450 and EFS 300 shared storage solutions provide advanced project sharing capabilities including Avid-style bin-locking, project-sharing and multi-user write access to media spaces for its Adobe Premiere Pro and Avid Media Composer editing systems. At UWE, student project collaboration is a must, making the EditShare XStream EFS project sharing feature a critical component. Student media and projects can be stored in the same Media Space, or in separate Spaces with files are owned by the members of the Media Space –including media files, media database files, project settings and bins –and any user with permission can modify or delete the files.

For more information on EditShare solutions, please visit www.editshare.com or to read the full case study on UWE Bristol, please visit www.editshare.com/case-studies/cs-uwe

About EditShare

EditShare is a technology leader in networked shared storage and tapeless, end-to-end workflow solutions for the post-production, TV, and film industries. Our ground-breaking products improve efficiency and workflow collaboration every step of the way. They include video capture and playout servers, high-performance central shared storage, AQC, archiving and backup software, media asset management, and Lightworks – the world’s first 3-platform (Windows/OS X/Linux) professional non-linear video editing application.

©2018 EditShare LLC. All rights reserved. EditShare is a registered trademark of EditShare.

Press Contacts

Alex Molina
Zazil Media Group
(e) alex@zazilmediagroup.com
(p) +1 (978) 866 7354

BOSTON, Mass.—At NAB Show New York EditShare will showcase its EFS scale-out intelligent shared storage solutions and Flow media asset management, which is now compatible with third-party storage.

The XStream EFS storage family of products allows collaboration with creative editing/compositing and grading solutions and features native drivers for Windows, Mac and Linux platforms, mitigating performance degradation compared to standard IT transactional hardware that leverages sharing protocols such as SMB or NFS.

New capabilities for XStream EFS storage solutions emphasize security and accountability. The company says the EFS XStream Auditing dashboard is the industry’s first purpose-built scale out storage solution with auditing designed for the media and entertainment market and to deter cybercriminals by providing system administrators with a digital footprint of all file interactions.

The new software-defined Flow, now available, has been re-engineered for non-EditShare storage environments such as Avid NEXIS, StorageDNA and Amazon S3. Bringing search, browse and retrieve capabilities as well as tools to assemble media to static media storage, the new software-defined MAM platform maximizes the value of customers’ existing storage infrastructure by adding an intelligent media management layer that the company says can manage millions of assets across multiple storage tiers in different locations.

EditShare will be on the show floor in booth N332.

[Want more information like this? Subscribe to our newsletter and get it delivered right to your inbox.]

Basingstoke, UK — August 30, 2018 - EditShare a technology leader in intelligent scale-out storage, AQC and media management solutions, announced today that Cape Town, South Africa-based Magnatude Facilities, a division of Okuhle Media Group, has selected an XStream EFS 300 scale out storage platform, EFS 40NL nearline storage nodes and Flow media asset management to overhaul their media entire workflow.

Prior to implementing EditShare workflow solutions, the team worked off an aging RAID setup that wasn’t equipped to support media intensive workflows, often slowing down with as few as three editors working on a project. In addition, there was no way to search for or retrieve existing assets. As a result the team would lose valuable time and money as a consequence of having to reshoot material they already knew they had- they just couldn’t find it.

“Implementing Flow has completely changed the way we interact with and search for existing footage,” says Ryan Sheraton, live director and post production supervisor at Okuhle. “For example, we’ve shot Table Mountain for nearly every single show that we’ve delivered and broadcast because we didn’t know where our existing iconic shots were. (Table Mountain is a prominent landmark overlooking the city of Cape Town in South Africa). Now with Flow, we can basically just type in ‘Table Mountain’ and the raw footage appears. It’s been an incredible time saver and we don’t waste time and money going out there to get that shot again.”

Faced with a major new project, Showville, a reality television talent show in the vein of America’s Got Talent!but focused on the smaller areas of South Africa, the facility’s existing storage infrastructure needed a major upgrade. With a production crew of 20 deployed to each location, the team knew that they needed a way to optimize their workflow in order to truly collaborate on this project and manage the huge amounts of incoming media.

“It’s one of the bigger shows that we have brought on as a client. There’s no way we could have taken it on with our old setup- it would have been a collaboration nightmare!” Sheraton continues, “With the EditShare systems in place we’re able to have four editors working on the show simultaneously. It’s a great achievement in terms of managing data and managing at least five cameras and four editors working in tandem.”

The production team shoots on location then the footage is couriered over to Magnatude Facilities on hard drives where it is ingested to the 40NL. The editors work with the raw footage off of the 40NL directly, easily accessing project files at the same time and collaborating to yield a final cut which is migrated to the production centric EFS 300.

“When we were working off our legacy RAID system, there was no way for us to share and work collaboratively on projects. As a result we literally had editors running up and down the halls asking people to close files, it was completely inefficient.” Sheraton adds, “Working on ‘Showville’ in this way would be impossible, the bottlenecks it would have created would have been a major blow and would introduce risk to on-time delivery. With the EFS 300 production platform coupled with the nearline storage 40NL system, everyone can open any project when they need it without compromising system performance, it’s awesome.”

Sheraton concludes, “EditShare solutions have given us the confidence to approach larger and more complex projects, we know these systems can take the heat. Going from what we called ‘Frankenstein’ to a reliable and robust storage and media management ecosystem has completely revolutionized how we approach our work. In addition, EditShare’s technical support team is incredible, they’ve always been available to us as an invaluable resource for troubleshooting and workarounds.”

“Successful post facilities like Okuhle need platforms that they can grow their business on,” comments Tara Montford, general manager, EditShare. “EditShare EFS allows facilities to step into an enterprise workflow with a ‘right sized’ infrastructure that maps back to their production needs, with the flexibility to expand on the fly. It's the best of both worlds. They invest in only what they need today with the freedom to grow the platform alongside the business.”

About the EditShare Workflow at Magnatude Facilities

EditShare XStream EFS is a powerful distributed scale-out video storage platform developed for the most intensive media workflows. It's designed from the ground up to support resilient large-scale workgroups and high-bandwidth, high-volume media ingest, transcoding, online collaborative editing and multiplatform distribution of HD, 4K, 8K and beyond. Administrators can easily add one or more XStream EFS 40NL nodes of cost-optimized storage to an existing XStream EFS storage cluster to create a nearline storage workflow like Magnatude Facilities or configure the EFS 40NL as an independent storage cluster for disaster recovery or media backup purpose, with each 4U storage node providing 240TB of raw capacity. EditShare Flow seamlessly integrates with EFS, providing a control layer that manages assets and metadata from ingest to playout with tools to search, retrieve and storyboard media, facilitating a collaborative post production workflow for Okuhle.

For more information on EditShare XStream EFS 300, EFS 40NL, and Flow media asset management, please visit: www.editshare.com.

About EditShare

EditShare is a technology leader in networked shared storage and tapeless, end-to-end workflow solutions for the post-production, TV and film industries. Our groundbreaking products improve efficiency and workflow collaboration every step of the way. They include video capture and playout servers, high-performance EFS central shared storage, AQC, archiving and backup software, media asset management, and Lightworks – the world’s first three-platform (Windows/OS X/Linux) professional non-linear video editing application.

©2018 EditShare LLC. All rights reserved. EditShare is a registered trademark of EditShare.

Press Contact

Nick Govoni
Zazil Media Group
(e) nick@zazilmediagroup.com
(p) +1 (978) 866 7354

Boston, MA – May 10, 2018–EditShare, a technology leader in intelligent shared storage and media management solutions, will be exhibiting at the BroadcastAsia 2018, held in Singapore from June 26-28 on stand 6J4-04. Headlining the showcase are EditShare’s new XStream EFS scale-out, high-availability shared storage, QScan automated quality assurance and new Flow software defined media asset management solutions. “In terms of content, the Asia creative market is generating world-class programmes and films that rank among the most watched on the planet,”comments Peter Lambert, sales director, EditShare. “The facilities behind these global blockbusters are employing advanced workflows and EditShare is here to support those requiring more comprehensive content protection and quality assurance to meet the international delivery demands for their popular content.” India-based Red Chillies recently invested in EditShare scale-out shared storage to support its upcoming “Zero” production starring Shah Rukh Khan. Lambert adds, “When you are supporting productions withmega starslike Shah Rukh Khan, you have to have the very best production infrastructure in place. Zero compromises and this is what EditShare delivers for its customers.” EditShare will be showcasing a number of new advancements across security, compliance and remote workflows including:

Scale-Out Storage with Industry Best Practice-Compliant Content Security Capabilities
The new EditShare XStream EFS storage is the first media purpose-built storage solution that supports comprehensive File Auditing. The new XStream EFS File Auditing platform allows users to collect and analyze shared storage user activity. In addition to forensic purposes, data can be used to assure clients that their content is stored according to MPAA best practices, meeting security audit protocols and procedures. Detailed reports pinpoint which user carried out an action, what files or directories were affected, and exactly when these actions took place. Unlike other auditing approaches, EFS Auditing does not impact real-time operations or system performance. EFS Auditing data is easily exported to third parties and other auditing applications.

Ensuring Delivery Compliance Through Automated Quality Assurance
Powered by the award-winning QUALES QC engine, the new file-based video and audio AQC QScan product line simplifies compliance and delivery requirements with robust quality check capabilities that leverage a patent-pending QScan Single-Pass Analysis process, which can be applied at any point during the workflow. Certified by DPP and AMWA, all EditShare QScan models support DPP compliance, including PSE and IMF testing. Models include:

• QScan One – A robust full-featured entry-level AQC solution that processes one file at a time. Ideal for small post facilities requiring the facility to check the integrity of files on an on-going basis but that don't need to check large volumes of media concurrently.
• QScan Pro – A full-featured professional AQC solution that processes four files at a time. Designed for mid-sized post facilities, with different departments such as audio, grading, VFX and editing. Each department can set up parameters to test their files, with up to four files being tested simultaneously.
• QScan Max – A fully scalable multi-node enterprise AQC solution with each node allowing for processing four concurrent files at a time. Ideal for VOD, OTT and Telco companies, the Max revision has limitless scalability, allowing a large operator to test hundreds of files concurrently.

Software Defined Flow Media Asset Management Powers Remote Workflows
Designed for non-EditShare storage environments such as Avid NEXIS, storageDNA and Amazon S3. As a fully software-defined MAM platform, Flow has been completely re-engineered to maximize the value of customers’ existing storage infrastructure by adding an intelligent media management layer that can manage millions of assets across multiple storage tiers in different locations. At the core of Flow lie several workflow engines that enable collaboration through Ingest, Search, Review, Logging, Editing and Delivery, alongside a powerful workflow automation engine for automating tasks such as transcoding and delivery. Flow’s award-winning remote workflow features also provide the ability to review content remotely, as well as edit content on a timeline with voice-over and effects from anywhere in the world.

Along with over 500 software updates, this new version of Flow features a redesigned UI, unifying the user experience across web-based and desktop apps. Flow Story and AirFlow now have a similar look and feel, making it easier to switch between them in different scenarios. Flow has also introduced new capabilities to remotely view Avid Media Composer or Adobe Premiere edits in a web browser using AirFlow; Range Markers for enhanced Logging and Review capabilities; new software licensing with a customer portal and license management tools; and seamless integration with EditShare’s new QScan AQC software, making AQC available at any stage of the post-production workflow.

The demand for remote workflows has also increased over the past year, as more people work outside of the traditional post-production environment. Out of the box, Flow caters for this increasing trend by enabling full remote access to content, as well as seamless integration with leading NLEs such as Avid Media Composer and Adobe Premiere.

About EditShare
EditShare is a technology leader in networked shared storage and tapeless, end-to-end workflow solutions for the post-production, TV and film industries. Our groundbreaking products improve efficiency and workflow collaboration every step of the way. They include video capture and playout servers, high-performance central shared storage, archiving and backup software, media asset management, and Lightworks – the world’s first three-platform (Windows/OS X/Linux) professional non-linear video editing application.

©2018 EditShare LLC. All rights reserved. EditShare® is a registered trademark of EditShare.

Press Contact
Nick Govoni
Zazil Media Group
Email: nick@zazilmediagroup.com
Tel: +1 (978) 866 7354

Basingstoke, UK — April 5, 2018 - EditShare a technology leader in intelligent shared storage, QC and media management solutions, today announced its flagship EditShare XStream EFS shared storage solutions will now feature ACL Media Spaces as well as enhanced storage space utilization reporting. “The big attraction of ACLs is the fine-grained control over user permissions. While EditShare has traditionally focused on keeping storage management tools simple for creative users, 'power-user' administrators who are more accustomed to SAN storage and Windows-NT style controls will appreciate this new addition,” comments Bill Thompson, EditShare Storage Product Manager.

Along with the new ACL Media Spaces, EditShare has revamped the way it reports used and free space on its EFS systems. “Because EFS administrators can choose between several different file protection schemes, each requiring different amounts of storage capacity overhead, showing how much physical disk space is being used by a set of files, and reporting remaining capacity, can require some tedious mental gymnastics,” says Thompson. “Our new reporting package dramatically simplifies the situation.”

EditShare XStream EFS ACL Media Spaces and new storage space reporting capabilities will be demonstrated at NAB 2018 (booth SL8620). Attendees to the 2018 NAB Show can book a private demonstration with an EditShare expert to discuss shared storage workflow needs and how EditShare XStream EFS solutions can help at: editshare.com/book-demo-nab-2018.

New EditShare XStream ACL Media Spaces

The new ACL Media Space feature satisfies organizations who prefer to manage storage via Access Control Lists instead of "read-only" or "read/write" rules that govern current EditShare EFS media spaces. ACL Spaces are managed in Finder and Windows Explorer extensions that are installed with EditShare Connect. Users with the ACL Management Limited Administrative permission can modify ACL Space permissions. A future release will include a "Full Control" attribute which gives non-limited Administrative users the right to set ACLs where they have been given Full Control.

Key ACL Options

• ACL entries can be created for local users and groups, as well as Active Directory users;
• ACL entries govern Read, Write, and Execute access;
• ACL entries on directories can be set to be "Inherited" or not. If they are inherited, all files and sub-directories made inside that directory will inherit the same ACL;
• ACL entries can be applied on a single object (file or directory), on all files in a directory, or recursively down to everything inside and underneath a directory;
• ACL entries can be added, modified, or removed for one or more users at a time;
• Users who have ACL-setting authority can take ownership or change the group of a file or directory;
• It will be possible to create "Drop Boxes" where users can write files but then can't see what they wrote — or what anybody else wrote.

New Improved Space Utilization Reporting

Before the release of the EditShare XStream EFS scale-out storage solutions, it was simple to view how much space was being used on the EditShare storage systems, and how much space remained. The new EditShare XStream EFS release brings back that simplicity offering a visual easy-to-read dashboard that summarizes available usable space whether you are using traditional media spaces or new ACL managed spaces.

About EditShare

EditShare is a technology leader in networked shared storage and tapeless, end-to-end workflow solutions for the post-production, TV, and film industries. Our ground-breaking products improve efficiency and workflow collaboration every step of the way. They include video capture and playout servers, high-performance central shared storage, archiving and backup software, media asset management, and Lightworks – the world’s first 3-platform (Windows/OS X/Linux) professional non-linear video editing application.

©2018 EditShare LLC. All rights reserved. EditShare is a registered trademark of EditShare.

Press Contact

Nick Govoni
Zazil Media Group
(e) nick@zazilmediagroup.com
(p) +1 (978) 866 7354

Boston, MA — March 8, 2018 - EditShare a technology leader in intelligent shared storage and media management solutions, today announced that it will be showcasing new capabilities across its award-winning XStream EFS scale out storage, Flow media asset management, QUALES automated quality check and file verification, and XStream EFS 40NL nearline storage solutions at NAB 2018 (booth SL8620). Attendees to the EditShare booth will see first hand how the new XStream EFS Auditing Dashboard mitigates potential hacking and content theft, as well as how XStream EFS 40NL enhances tiered storage workflows. EditShare media experts will also demonstrate QUALES AQC capabilities as integrated with Flow Media Asset Management, revealing new workflows that lower costs and meet the demands of evolving delivery requirements for popular OTT platforms. Finally, EditShare will be showcasing for the first time how the brand new, software-defined Flow media asset management platform expands remote workflow opportunities and collaboration.

“You can be producing outstanding work, but if your content is not secure and your delivery methodology is spotty, you are putting your business at risk,” says James Richings, Managing Director, EditShare. “With the expansion of IT into media workflows and the growing popularity of OTT platforms, the time is now to ensure your facility has the right security layers and delivery methodology in place to minimize the risk of stolen content and bad file format deliveries. EditShare’s NAB showcase presents new storage auditing, media asset management and quality assurance solutions that address workflow weaknesses with best practice solutions that align with industry standards that will soon become table stakes for doing business.”

Attendees to the 2018 NAB Show can book a private demonstration with an EditShare expert to discuss their workflow needs and how EditShare solutions can help at: editshare.com/book-demo-nab-2018.

XStream EFS File Auditing Capabilities

At IBC2017, EditShare unveiled EFS XStream Auditing - the industry’s first purpose-built scale out storage solution with auditing designed for the media and entertainment market. This revolutionary product provides comprehensive auditing and reporting of all file system activity, including file creation, deletion, modification, opens and reads as well as general media space access. EditShare auditing capabilities are designed to deter cyber criminals by providing system administrators with a complete digital footprint of all file interactions.

Unique to this solution is the intuitive XStream EFS Auditing Dashboard that provides administrators a high-level view of activity with the ability to drill down to individual users and files.

EditShare EFS 40NL Nearline Storage for EFS Tiered Storage

The XStream EFS 40NL delivers the performance and economics of traditional nearline storage with the scalability, fault-tolerance and ease of use of the proven EditShare EFS shared storage solution. Administrators can easily add one or more EFS 40NL nodes of cost-optimized storage to an existing XStream EFS storage cluster as well as configure the EFS 40NL as an independent storage cluster for disaster recovery or media backup purposes.

QUALES Automated Quality Check & File Verification

Now AMWA-compliant for AS-11 UK DPP, the EditShare team is paving the way for easy compliance at the the click of a button. The unique QUALES user interface is designed for the creative, giving an instant visual presentation of the quality-checked file. Flow users, which include producers, post supervisors, editors and other non-engineer staff, will have an intuitive visual aid to spot-check content. With QC offered as a core capability within the EditShare Flow media asset management platform, users will be able to systematically QC content at any point across the workflow saving countless hours of production and post-production time.

Stay tuned for a massive QC product lineup launch on the NAB Show floor.

Flow: Independent Software-Defined Media Asset Management Platform

Announced at IBC2017, Flow will be available as a storage-agnostic software solution and will work with industry-standard storage solutions, providing a layer of content creation, media management and workflow automation that optimizes the entire production workflow. Search, find and retrieve valuable assets quickly and efficiently, while getting content directly into your favorite NLE, whether it’s Adobe Premiere, Avid Media Composer, Apple Final Cut Pro or Lightworks.

In addition to the new solutions, EditShare will have its full lineup of products on hand including all scale out shared storage models XStream EFS 450, EFS 200 and EFS 300 scale out storage; and GEEVS multi-channel ingest and playout server.

About EditShare

EditShare is a technology leader in networked shared storage and tapeless, end-to-end workflow solutions for the post-production, TV, and film industries. Our ground-breaking products improve efficiency and workflow collaboration every step of the way. They include video capture and playout servers, high-performance central shared storage, archiving and backup software, media asset management, and Lightworks – the world’s first 3-platform (Windows/OS X/Linux) professional non-linear video editing application.

©2018 EditShare LLC. All rights reserved. EditShare is a registered trademark of EditShare LLC. All other trademarks mentioned herein belong to their respective owners.

Press Contacts

Nick Govoni
Zazil Media Group
(e) nick@zazilmediagroup.com
(p) +1 (978) 866 7354

Flash memory provides a needed level of storage performance for media and entertainment applications which now command, for UHD and beyond, massive amounts of storage and speed to meet the growing amounts of content being generated. New high-res applications demand much higher storage performance than conventional enterprise back office applications. Flash memory is helping enable that performance—but with it, the physical media also commands interfaces that provide the best value for the applications and its associated storage.

Non-volatile memory express (NVMe) is a trending technology that utilizes Flash storage more effectively and efficiently. The harmony of solid state and rotating storage media will continue for the near term, and for the undefined future. These respective storage mediums will endure, providing complimentary values for ambitions such as more storage, better storage, and faster throughput with a reduced hardware footprint, at less cost.

[Read: A Solid State of Non-Volatile Memory]

MORE THAN A 2X BUMP

At the previous two Flash Memory Summits (2016-17) both NVMe and the PCIe 4.0 bus were hot topics. Yet right alongside the 2016 Flash promotors, the SCSI Trade Association reminded the industry that “a new serial-attached SCSI (SAS) technology was on the way.” That new technology took the SAS ecosystem from the 12G level to a usable 24G SAS (24GBps, serial attached SCSI). Promoters unveiled this advancement as more than just a “two times bump in speed over the previous data rates for 12G SAS” (refer to Fig. 1 for the SAS technology roadmap).

The interface of choice for mission-critical storage applications remains SAS; moving the interface from 12GB to 24GB yielded a major refurbishment in the technology. Updates included more efficient 128b/150b encoding, with SAS Protocol Layer (SPL) packets and Forward Error Correction (FEC). The transmission signaling rate is specified at 2.4GBbs (i.e., 22.5 gigabaud rate) which still retains compatibility with earlier 6G and 12G solutions.

Changes which help achieve this faster throughput include an FEC field, 20 bits in the SPL packet that aid in error detection and recovery. An SPL packet is a 150-bit block that includes a 2-bit header and a 128-bit packet payload, plus the FEC bits. At the deep-dive level, additional features added include changes in: binary primitives (in the SAS Link Layer); primitive parameters; serial management protocol (SMP); open priority; and inter-expander fairness arbitration enhancements. Details can be found in a technical overview document for Serial Attached SCSI, which is roughly 1,000 pages and whose details are beyond the scope of this overview.

LOWERING LATENCY

Regardless of the application for media and entertainment or others such as transactional trading on the stock exchange, latency has a tremendous impact on storage performance. Recalling from our previous article—NVMe is an industry standard that is optimized with a new storage stack featuring a low latency, efficient and scalable protocol streamlined with a revised drive command set that uses fewer clock cycles per IO operation.

Comparing spinning magnetic drive latencies for hard disk drives (HDD) to those of Flash SSDs, conventional Flash NAND technology offers a 100x reduction in latency over HSDs.

Latency figures decrease further when NVMe eliminates the 20 microseconds of latency found in the SSD NAND (whether SAS or SATA) implementation. “Next Gen NVMe” will now drive NVMe to deliver “4KB operations in under 10 microseconds,” according to a presentation at the 2016 Flash Memory Summit.

CHANGING THE MESSAGING

What do these memory improvements provide to the media and entertainment (M&E) industry? For starters, it enables audio/video to become the dominant medium for the carriage of information and content. In a February “New York Times” text message to mobile subscribers we saw, “What you are doing now (i.e., ‘reading text on a screen’) is likely going out of style.” The Times’ text infers that A/V will replace the text messaging we do today and in the not-too-distant future.

One of the tasks necessary to achieve this change will be to increase the speed of the delivery while at the same time improving the processing and memory requirements of the channels which deliver the information. The new 5G networks may help this, but more is needed. This same prolog can be applied to production and post-production workflows for M&E.

[Read: Non-Volatile Memory Grows In Popularity]

We have already seen the impacts of 4K over HD in terms of resolution, quality and image perception. While well-produced 1080p content can be wonderfully upscaled for 4K displays; in many cases there can be very satisfying results when the content is originated as 4K and then downconverted to 1080p and displayed on a high-end HD (1080p60) television system.

The real impact for UHD/4K, once available in more delivery systems, will be in the use of higher frame rates (1/120 second frames instead of 1/60 second) especially for sports. Add the higher resolution per frame and a higher density of pixels per unit screen area, and it could make today’s conventional HD images look like older analog 1-inch videotape, comparatively.

IT TAKS A LOT OF MEMORY

To reach these goals, it takes memory, a lot of it; and fast memory coupled with much higher bandwidth (selected video formats described in Fig. 1). Compression helps to a degree, but with the improvements in compute and networking technologies, more productions are switching to full-bandwidth, uncompressed video for their editing and post-production workflows.

Digital storage for media workflows will continue to advance in these areas. Flash, coupled into NVMe interfaces, make these higher capacity, faster and better memory solutions possible. We’ve seen Seagate produce a 12TB “BarraCuda Pro” 3.5-inch HDD with SSD-like performance, and SanDisk’s “Extreme PRO” CFast 2.0 solution in a 256GB form factor capable of 525MBps read (450 MBps write). These devices are the tip of the iceberg, as the products become the workstation’s local drives (i.e., the Seagate HDD) and the camera capture’s memory solution (i.e., the SanDisk memory)—key components needed to work in higher resolution and higher frame rate production workflows.

ACHIEVING THE TARGET

Along with this perspective on higher resolution, higher bandwidth production—another element which will consume massive amounts of memory is video on demand. According to Coughlin Associates, the shipping capacity for VOD storage will increase 8-times, from the 2,500,000 TB (terabytes) in 2016 to a whopping 20,000,000 TB (per year) in 2021. Add the other data sets, which aren’t getting any smaller, and the requirements necessary to fulfill all these ambitions is quite an undertaking. When AI and VR take hold, cloud services may not be enough to satisfy such exponential changes in storage volumes—let alone the means to transmit (wirelessly) the demands to the end users.

To manage the physical maintenance issues of storage using HDDs could require small armies of support teams (quite possibly robotic) just to change the drives in even a modest data center. Flash memory would need less support, with the R&R (rescue and recovery) cycles for SSD upkeep probably hundreds of times less. Technological changes would likely outpace the life of the SSDs in service, making storage a disposable commodity.

This is the future for storage. Mechanical drives, besides reaching physical capacity thresholds that may limit their overall performance, are probably in their last decade of useful life—at least for large scale M&E solutions. Single unit workstations may continue with HDDs, where cost-to-capacity requirements are different than in datacenters—although the migration to all SSDs is very much apparent in tablets and mobile devices. Watch for some dramatic changes in storage and Flash as more video enters into our lives and better images become an expected case and not a “luxury once in a while” alternative.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Click on the Image to Enlarge
Click on the Image to Enlarge

Even as a mature, diverse and reliable technology—magnetic spinning disk drives (aka hard disk drives or “HDDs”) continue to grow in capacity, performance and cost benefits. Alternative storage solutions, however, continue to evolve. Hot on the HDD heels—as has been the case for more than 10 years running—are a family of solid-state equivalents.

SSDs (solid-state drives) now sit squarely alongside the other legacy nonvolatile memory solutions, and their presence is being enhanced by interface improvements known generically as non-volatile memory express or NVMe. 

Storage media itself, such as Flash Memory, have seen a multitude of improvements centered on many technological advances. My previous columns have outlined those changes over the past decade. Now we see new steps to improving non-volatile memory solutions, which are in the interfaces themselves.

HDD EVOLUTION

Of the various interface forms for HDDs, those with serial Advanced Technology Attachment (SATA) interfaces

have become the more cost-effective and most prominent of the “everyday application” disk drives. The more expensive serial-attached SCSI (SAS) drives buy the users other capabilities.

When it comes to HDD applications, SAS drives tend to be found more in enterprise computing because of their high speed and high availability, factors crucial for such activities as ATM transactions, stock exchanges and eCommerce. 

Conversely, SATA drives are used primarily in desktops for consumer use and in those less demanding roles such as backups and near-line data storage.

We are omitting Fibre Channel disk drives from this conversation because they are more specialized and less cost effective, but can arguably be justified in high-performance storage applications (such as editing systems) when supported by the appropriate operating and file system technologies.

IOPS AND RELIABILITY

Keep in mind that the best measure for HDD speed is IOPS (inputs/outputs per second), specifically when the drives are in use, under stress and with real applications designed to optimize the drives’ capabilities.

To put IOPS into perspective, industry-accepted averages for 7.2K SATA drives is about 80 IOPS, with the 10K (RPM) devices offering around 120 IOPS and 15K pushing the limits of around 180 IOPS. The equations turn dramatically for solid-state storage devices (SSS), with huge IOPS improvements and no mechanical worries. 

The other factor for SAS v. SATA is reliability. The industry-acknowledged MTBF (Mean Time Between Failure) for SAS HDDs is around 1.2 million hours, compared with 700,000 hours MTBF for SATA drives. This also places SAS clearly into the enterprise space.

Once you move from the HDD world to the SSD world and to other forms of non-volatile memory, the feature sets and interfaces begin to shift. Not only will speed (performance) increase, but the applications for which SSDs can be applied to help increase total system performance—not just from the storage I/O perspective.

Table 1: Some familiar, but less recognized non-volatile memory (NVM) types.

Non-volatile memory (NVM), sometimes called NVS (non-volatile storage), is a classification for a form of digital storage (memory) that retains its state without having power continually applied. Generally, this storage media is without any mechanical components, although that is not necessarily the case.

Optical storage is considered in the NVM classification, as would be any readonly storage that doesn’t require electrical stimulus to retain its state (e.g., a PROM/EPROM). See Table 1 for examples.

Initially, NVS and NVM were intended for secondary storage or for other longterm persistent storage mediums. Today, in its SSD format, it is often used for primary storage to support short-term RAM/DRAM—as in laptops, tablets or mobile devices.

INTERFACE EXPRESS

Besides the physical storage media, electrical and software interfaces are needed to support the “NVM-storage” term. Common interface methodologies include Non-Volatile Memory Express (NVM Express or NVMe) and NVMe over Fabrics (NVMe-oF or NVMeOF). See Fig. 1 for an example of one method for the physical interface.

The “NVMe” term mystifies many, with some feeling these terms are becoming more hype than practicality. We hope to provide some clarification with the following.

NVMe is a host controller interface and storage protocol established to accelerate the data transfer between host/enterprise or client systems and solid-state drives over a computer’s high-speed PCIe (Peripheral Component Interconnect Express) bus. NVM Express (v1.3) is an open collection of standards and information that exposes the benefits of nonvolatile memory (NVM) in computing environments from the mobile device to the data center. The specification, and its registered and trademarked explanations, can be downloaded from the NVM Express Inc. website (www.nvmexpress.org). The term “NVMe” is a trademarked name, which encompasses a solution set designed, from the ground up, to deliver high bandwidth and low-latency storage access for NVM technologies.

The NVMe specification defines a register interface, command set and collection of features for PCIe-based SSDs. Its goals are to enable high performance and interoperability across a broad range of NVM subsystems. Note that, like most “standards,” the NVMe specification does not stipulate the ultimate usage model, i.e., how NVMe is directly associated with a specific solid-state storage, main memory, cache memory or backup memory. 

NVMe is optimized for Enterprise and Client solid state drives, typically attached as a register-level interface to the PCI Express interface. The 287-page specification describes an interface that allows host software to communicate with a nonvolatile memory subsystem. During its development, the specification was referred to as “Enterprise Non-Volatile Memory Host Controller Interface Specification (NVMHCI).” The lengthy name was simplified to NVM Express prior to publication.

Fig. 1: PCIexpress interface card with NVMexpress solid-state drive unit fitted to slot on the interface card. Card uses an M-Key edge socket to attach the SSD to the PCIe form-factor adapter card.

The NVMe 1.3a spec addresses both NVMe over PCIe and NVMe over Fabrics. The later Fabrics specification defines a protocol interface and related extensions to NVMe that enable operation over other interconnects (e.g., Ethernet, InfiniBand, Fibre Channel). Support requirements for features and functionality may differ between NVMe over PCIe and NVMe over Fabrics, rendering different performance parameters based upon the interface application.

NVMe increases support for Enterprise capabilities via enhanced error reporting and virtualization. End-to-end data protection is compatible with SCSI Protection Information, known as “Data Integrity Field” (DIF). DIF (or T10 DIF) is an approach to protect data integrity in computer data storage from data corruption, originally proposed in 2003 by the T10 subcommittee of the International Committee for Information Technology Standards (INCITS). SNIA references this as “Data Integrity Extension” (DIX) in its standards.

KEY FEATURES

The NVMe 1.3a spec interface has many additional key attributes, such as an efficient and streamlined command

set and support for multiple namespaces and namespace sharing. The NVMe Management Interface is the command set and architecture utilized in out-of-band management of NVM Express storage (e.g., discovering, monitoring and updating NVMe devices using a BMC). Typically, the NVM Express controller is associated

with a single PCI function.

In similar fashion to the transitions of HDD interfaces that went through ATA, IDE, SCSI and beyond, now, when you think about solid-state drive technologies, you can add the latest dimensions of the NVMe interface—one of the hotter topics in storage technologies that are growing stronger each day.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. Read more about storage topics in his book “Moving Media Storage Technologies.” He can be reached at kpaulsen@diversifiedus.com.

In my last column, we looked at the fundamentals of object storage as it applies to the media industry and as applicable to archiving. Object stores have become a principle solution for long-term data preservation, especially for cloud-based environments—whether for on-prem or private clouds. Object storage is used heavily in public cloud storage solutions and especially when the data is geographically disbursed for protection and accessibility purposes.

Furthermore, where tiered storage has been a trend for more than a decade, object storage brings new perspectives—particularly when addressing disk replacement management. Object storage may now be overshadowing even some of the earlier approaches to tiered storage. 

A comparison of how traditional versus object storage is implemented

Tiered stores were, for many years, an integrated overall approach to having different drives and physical media applications arranged in such a way that high-performance drives (those at the highest/top tier) were used for immediate access by applications; and lesser performing storage solutions (e.g., tape) were used for longer-term storage.

HIGH-PERFORMANCE, TOP-TIER STORAGE

Applications, for example those for post-production video editing or graphics compositing, require data delivery to be extremely fast and do so with minimal latency. Often these top-tier storage devices were Fibre Channel based (i.e., both the drives and the network switch configurations are Fibre Channel). These systems would be coupled with specific file-system controllers (aka “metadata” servers) which are designed to manage the high-volume/high-throughput transfers necessary for very rapid data movement to and from the base system application servers, local workstations or the compositing, rendering or effects servers.

The mid-tier storage was often referred to as “slow-disk” or “near-line” storage. The content data held at this level were usually the completed works (finished edits), as well as complimentary versions of the first master completed clips, stories or commercials. In addition, various “b-roll” pieces might be held in the same near-line storage since fast accessibility was not necessarily needed during content approval periods or lulls during the post-production periods. Often this storage tier was also used as the holding location for content which would ultimately be readied for archive or much deeper/long term storage.

The drives needed in this application need not be expensive, top-tier level (fibre channel) drives. Even the types of drives used in consumer-level PCs or laptops would suffice for this level of storage.

ARCHIVE NEEDS AND CONSIDERATIONS

The lowest tier storage was usually a linear tape storage solution or possibly cloud storage. If stored to tape, sometimes the workflows would require a local copy in a tape library be retained, and an off-site copy (e.g., an “iron mountain” version) would be created and physically shipped to another location.

It is this lowest tier, the archive tier, which is steadily being replaced by the disk-based object storage solution. The reasons for using object storage vary depending upon the current investment the archive-provider may already have in tape-based storage; or how the organization feels about shifting to newer technologies (i.e., disk storage either on-prem or offsite); or if the organization feels tape storage is too costly or they’ve never used tape-based library solutions and don’t want to pay for the costs of private or public cloud for archive.

In the latter case of cloud storage costs, this rationale will depend upon what form of the cloud you choose or if the potential time line needs for data restoration (recovery) cannot be predicted. These factors change the equation in terms of both the cost for data recovery (i.e., getting the data back to your own mid-tier storage) or how quickly, time wise, you might expect that data be needed since some archives use storage which cannot be easily just called up and returned to the user.

Deep archives are meant to take data in at very low costs and charge you much more to return that data. The more quickly you want the data recovered, or the volume of the data you need plus the time you need it back can seriously impact the costs for storing (and recovering) that data. 

FAST DATA RECOVERY IMPACTS

Some organizations simply don’t know when they’ll ever want the data back or how fast (e.g., disaster recovery versus occasional return uses). For this reason, by itself, building an archive which allows for the rapid return of the data (whether from offsite or onsite tape or from the cloud) becomes a business-level trigger whereby you might want to consider object storage and exclude tape or cloud, except in a few situations.

Another capability inherent in object storage-based solutions is the cataloging methodologies utilized in the store itself. Metadata is key to search and paramount to getting just the right data back from any store. In most non-object cases, the metadata is held in an external media asset management system, which needs its own database, application servers, and integration into the entire storage and workflow solution set. 

When the key metadata can be held within the object (store) itself and be searchable or modifiable by an external solution (that is, the object’s indexer), then the necessity for an archive-centric MAM changes. In objects, which are essentially “wrappers” or “containers” that hold both the content data and the rich metadata together as a single entity; only keywords and locations need to be addressed by these external MAM-like applications. This reduces hardware and software overhead, and allows the systems to be rapidly accessible. In many cases the object store solution will take less physical space (a smaller footprint) compared to the tape library and cassette storage needed in more traditional archive applications.

LONG-TERM COSTS AND BUDGETS

While this point might be arguable depending upon when, what or how much the user has invested in a tape solution—the long-term costs for object storage, once implemented, can be reduced versus tape. Why? Because tape has a finite life (both physically and technically) that mandates the data be migrated from an older format to more current higher density formats. If you have a large library, this means the older tapes (which hold far less data) are or will need to be updated; that is, replaced with the most current solutions on a recurring basis. This replace and renewal process may be every few years. So, both the physical media (tapes) are replaced and the tape-drive mechanisms will also need replacement to leverage the newer, higher density/higher capacity tape storage solution. And don’t forget, if the physical media is damaged, the data is lost forever, unless a second copy is held elsewhere.

In object storage solutions, there will be multiple drives which (in similar fashion to RAID) protect the data across many sets of drives of that object solution. In some object store chassis, which may for example, consist of 20 to 40+ drives in each group set, as many as 4 to 10 drives can fail—and all the data can still be recovered. In large data centers (or clouds) disk drives are seldom replaced since the labor to remove and replace a single drive—per instance of failure—cannot be justified. When an object group gets to the point it becomes too high of a risk to implement data recovery (that is, maybe 6 of the 8 drives in a 20-drive group have failed); a decision can be made to update that chassis, or maybe retire the chassis altogether. 

MAKING BEST SENSE

Object storage makes sense when rapid access to data is necessary; when the risk to not being able to recover the data quickly is unwarranted for your operations; or if a cloud solution is not financially beneficial (coupled with either of the other two reasons). And since object storage systems allow you to replace older smaller drives (e.g., 2 or 3 TB drives) with higher capacity drives (mixing multiple size drives in the same group set or chassis), then the migration issues associated with tape vanish.

Next time you need to look at archiving content without the complications of tape or cloud, consider how an object-based system might fit into your organization’s workflows and budget.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl at kpaulsen@diversifiedus.com.

For some time we’ve thought mainly about how file-based storage is used to contain unstructured data; i.e., those files relative to moving (video) or static (photographic) images. File-based storage has traditionally been suitable for structured data—such as computer information when employed on personal computers or office workstations. This type of data can be appropriately organized, that is “structured,” for applications such as email, discrete sets of “office” documents, and project-related applications.

As “media”-related data came into the workforce, the storage of that data was also managed in the same way, despite the absence of true organization (or “structure”) to that data and no “best practices” for how to manage it. This type of media-centric data became known as “unstructured” data; a term which lives on through today.

In the early days of digital graphics (generated as individual files), photographic images (also individual files) or video (as contiguous groupings of inter-related files), few realized that digital media would expand to the degree it has in the past decade-plus years. Little work was done to address this eventual quantum shift in data storage needs and requirements. As linked sets of JPEG images for professional video media moved to streamed “strings” of compressed data, the volumes of files continued to grow in terms of both the formats and the quantities of actual content.

There were those who believed that asset management solutions might provide the needed organization of unstructured data. Those MAM-like processes however would still rely on traditional file-based storage solutions on the physical media (tape, hard drives, etc.), with content driving many new features and functions. Harddisk capacities increased, metadata became more important, better caching methodologies were developed, and transfer bandwidths grew to address the speeds and file-sizes of this new era.

When content creation exploded, driven by things such as digital cameras and personal mobile devices, and coupled with social media connections which now seem second nature, it became apparent that the overhead needed for file-based storage of media put huge bottlenecks in the processes of moving from one medium’s format to another. Another method for storage was needed—that methodology is called “object-based storage.”

Fig. 1: Advantages in utilizing object storage for unstructured data in archiving, cloud and geo-dispersed applications.

ALTERNATIVE STORAGE
Object-based storage, which is specifically formulated to address unstructured data, is the new alternative to file storage. This relatively recent “object-technology” is built around the concept of extended metadata and collected sets of associated data. An object can be thought of as a container—a “wrapper-like” entity that captures the unstructured data and its applicable bits-about-the-bits (aka “metadata”)—and houses it in a storage space that is designed for easier retrieval than file-based storage (Fig. 1).

In object storage, each object will be assigned a unique identifier that enables servers to retrieve it from any physical location. This is a core principle in object storage which is how cloud-based data is managed on a global basis.

Object storage systems—sometimes called “object stores”—can be software-only or hardware based. Smaller stores are typically hardware-based, but large data center size solutions will employ software-based solutions that allow the storage to be deployed on a much broader basis.

SCALABILITY
Object stores provide infinite levels of scalability, something that traditional file- or block-based storage systems cannot equal. Still there are challenges in making object stores work in a block- and file-based domain.

Interfaces continue to be a key component in making file- and block-based external shared storage successful. The two external shared storage system protocols (block and file) have flourished primarily because they are as widely used and are as available as the networking interfaces that drive them.

Traditionally, block-based storage solutions have utilized Fibre Channel and Ethernet (iSCSI) as their interfaces. For file-based solutions, the interface is usually Ethernet (e.g., CIFS/SMB and NFS).

Due to issues surrounding data protection, indexing, and addressing in large-scale data repositories, block and file (and RAID) are not very well suited for data center size storage applications. Disadvantages include RAID’s inability to scale sufficiently (and efficiently) and file-based protocols that run into issues with metadata management when the storage volumes approach petabyte-sized data and/or contain multiple billions of files. We add, however, that some enterprise-class storage providers have developed high-performance provisioning for billion-plus files or when capacity reaches or exceeds five or more petabytes.

Despite its advantages as an answer to storing data at the multipetabyte level, user applications still expect to see the functionality of the more traditional NAS or SAN interfaces. These needs complicate object store integration, making them less than straightforward compared with using block- and file-based systems. Nonetheless, object stores have taken off in popularity, especially in the last 2–3 years. Object store providers offer a variety of options for making this relatively new storage form work with key applications.

DISPERSED STORES
Object stores offer a means to provide dispersed (locally in the data center) and geo-dispersed (across countries and continents) data protection. Object stores do this without RAID, using protection mechanisms typically employing a form of erasure coding—otherwise known as forward error correction (FEC).

Lost or corrupted data can be recovered using a subset of the original content which, through algorithms, let the system reconstruct those lost storage elements mathematically; often in the background and with minimal disruption.

Erasure coding is far more scalable than RAID and is more efficient (time wise) although at a cost of additional CPU overhead. From a business continuity/disaster recovery (BC/DR) perspective, users benefit from erasure coding by allowing these “subsets” of erasure-coded data to be distributed geographically in distant locations or on adjacent floors (or buildings) on a campus-wide system. Object stores also offer failure protection capabilities, another feature leveraged when installations have more than one location for their storage platforms. Failure protection (as background rebuild task) is an erasure-coding feature void of the complexities, significant down time delays, and risks encumbered when having to rebuild RAID arrays employing multiterabyte hard disk drives.

Storage systems always run the risk of data loss or corruption. Very large-scale data repositories face this problem just like smaller systems. Most spinning disk and solid-state storage media are reliable, but not totally error-free. When storage media does fail, it may be because of silent (unknown) corruption or issues that result from unrecoverable read errors (URE). This obviously places all the data at risk.

SCRUBBING THE DATA
A technique called data scrubbing helps to validate and then rebuild potentially corrupt or missing data. Erasure coding algorithms along with the typical “write-once” (read many) nature of object store data enables failed data to be recreated in the background, with little or no impact to operations. This is another reason why object stores are becoming the first choice for long term, deep archives—as well as near term cyclical storage.

In a future article, we’ll take the next steps in discussing configuration, performance balances, and the chief advantages to an object store’s rich metadata management.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

No organization or individual is immune to the problems of limited storage capacity. However, many are finding new means to adapt their storage resources to alleviate the impact of not having sufficient storage to meet their operational and business needs. Some have moved to cloud storage for their archives or as mid-tier storage silos or for temporary “bursty” needs. Others have looked to virtualization technologies to help support cross utilization of both storage and compute resources.

Virtualization is bringing to industry new methods for the allocation of services across pools of resources ranging from servers (for compute and processing) to high-performance storage (for live work) or near-line (or deep) archive. Storage resource management processes help users make appropriate decisions. Applications vary from real-time storage environments to short-term “slow” storage to long-term “deep” storage for seldom-accessed archive purposes.

When organizations choose not to use public cloud services for storage or compute resources, they have likely either built their own “private” cloud or have already carved out a portion of their datacenter for on-prem storage. Storage management and virtualization can be applied to your on-prem resources in the same way as cloud solutions, but usually on a smaller scale.

DISSIMILAR STORAGE
No organization purchases “all the storage they’d ever need” at one time. The probability that there are differing sets of storage systems in any one datacenter or equipment room is high. In these cases, the groups of storage were probably acquired at different times, for different needs, and are most assuredly of different capacities, performance and drive denominations or configurations.

Fig. 1: Conceptualization of a storage system pool with surrounding components. The “pool” consists of various sets of existing disk arrays, which are combined into a single storage pool and managed through the virtual systems director.

Varying assortments of storage (see Fig. 1, yellow box) create sharing problems that aren’t easily managed by administrators.

In broadcast media and entertainment, post-production editing systems continue to scale in performance and demand. Recently storage and server vendors have made a renewed push toward uncompressed editing and to employ UHDTV/4K shooting and posting—driving storage capacities much higher than ever before. Any of these activities infer that storage systems will need continued updates with newer, faster arrays, and more capacity to support these higher resolutions.

The quandary of what to do with legacy storage systems—besides abandon it—is a question that has many factors to consider. Are the systems so old they can no longer be supported? Can they migrate older storage systems to other purposes when adding new, higher-performance storage systems to address new workflows? Is there sufficient life remaining on the existing storage system to retain value as a “pooled” resource?

Answers vary and need new considerations as storage management solutions make retention of existing or current storage systems more viable.

As capabilities in virtualization get easier to deploy, manage or become more cost-effective to implement, it makes sense to review opportunities for pooling of individual storage systems into consolidated sets of storage resources.

POOLS AND POLICIES
A storage system pool enables the grouping of similar storage subsystems as a managed, accessible set of storage, i.e., the “storage pool.” Using virtual systems directors (Fig. 1, green oval), users create a storage system pool of selected, available and appropriate storage subsystems. The virtual director, essentially a resource manager, lets authorized and authenticated users add storage to a storage system pool, permanently delete a storage system pool or edit a storage system “pool policy.” The storage system pool policy determines how the storage volumes within the storage system pool are allocated. Allocations are based on RAID level, bandwidth and storage pool preference settings.

Basic storage system pool management begins with discovering the available storage systems and resources—sometimes called the “audit.” Before beginning any automated (e.g., audit) process, users and administrators should be sure they’ve purged any latent files from the systems. MAM systems likely have garbage collection procedures built-in, which clean up duplicate, expired or scratch files. Conversely, simpler desktop editing systems usually won’t have this sophistication.

SCREEN AND ALLOCATE
Administrative procedures, whether manual or automated, should screen all the storage subsystems for any of the hundreds of file types, which may be written to servers or libraries—such as executables (.exe), MP3, GIFs, gaming or other files in quarantine. Once purged, the storage system pool process inventories of all the storage systems, reviews the applicable licenses (e.g., virtual machine control licenses installed or the time left in a complementary evaluation period, if applicable). If needed, the inventory process may allow users to purchase licenses for additional needed storage and may check for updates related to the management pool application.

Next comes the allocation process, which, once systems are appropriately configured, may be the manual segmentation of storage groups; or with a software-controlled environment, may be a “real-time” storage management solution provided by a vendor or a virtualization solution.

TEMPLATED OR REAL TIME
Storage in a virtual environment is usually managed in one of two ways: through the use of templates (or wizards), which will be “activated” in the future; and (b) in real time—where changes happen immediately or during the next restart.

When configuring storage using templates, you only create storage “definitions”—otherwise referred to as “configuration settings.” When creating “storage device definitions” as templates, no storage device is actually configured or altered. Before the definitions can take effect, they must first be “deployed.”

How the configurations are deployed depends upon how systems administrators have configured their deployment activities. By delaying deployments until qualified, any unintended activities, storage misallocations or those without authorization are prevented.

Conversely, when configuring in real time, the storage device will be active and connectivity would already have been established. All included devices would have to be discovered and any device the storage attaches to cannot be locked or made unavailable. When changes to the storage device are ready, the user opens a “panel,” which will make those changes immediately upon clicking “deploy” or “apply.”

AUTOMATED ONLINE STORAGE
When your internal/on-prem storage allocation reaches capacity, then alternatives will be needed. One alternative is “automated online storage.”

Traditionally, the managing of online storage was by “time interval scanning” of the overall workspace. Next the system performed automated checks to see how much storage remained, the amount of storage consumed during the last interval check, and included additional safety checks for unexpected situations.

Today, statistical data derived from such activities is pushed to integral databases and trending applications that watch current and past activities, then predict what may be coming. APIs may leverage third-party products, i.e., schedulers or calendars, to help analyze upcoming needs and predict future storage requirements.

With today’s unpredictable demands placed on storage, we must ensure there is sufficient storage to handle unexpected events or anomalies, which could occur between scans. Should a need for storage exceed the available allocated storage volume, resource managers will quickly add the needed storage—plus any overflow buffer—in real time. Calls to outside resource templates that stand “ready for deployment” may be enabled automatically, either on a local basis or through remote activation into the cloud.

Real-time “management implementation” may help reduce costs by mitigating the repeated environment scans and by providing up-to-date utilization analysis of data, which is then applied to maximize storage utilization rates.

Making the most of storage resources is now a routine part of overall systems management. Effective storage management requires that the correct technologies be blended with information technology practices applicable to the business’ activities (e.g., content collection and ingest, MAM, editorial and long-term productions or playout).

With the right tools, IT managers can then optimize the storage; plan steps to safeguard the data; and in turn, help reduce operational and long-term capital expenditures.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Storage systems, whether or not coupled with editing systems, MAMs or other production-related data systems within broad- cast facilities, are all headed in the direction of IP-based infrastructures.

Regardless of whether users are leveraging fibre channel or Gigabit storage, cloud technologies or simply augmenting their conventional on-premises systems in the central equipment room—the technologies now dominating the next-generation approaches to facility integration are becoming IP-based in every aspect.

Technologies for the use of IP in real-time media networks are rapidly moving into the production and routing infrastructure domains as we steadily progress towards implementation of the developing standards, including SMPTE ST 2110 and ST 2022-6.

WHAT’S LACKING?
What may generally be lacking in the “grand plan” for broadcast facilities is an ideal, consistent and interoperable network management system that can provide not only the control structures for studio video over IP (SVIP), but could become the logical extension to services like storage management, business process/workflow, direct uplink into the cloud and many other potentials.

There is still a lot to be accomplished in this area. Through the outstanding efforts and continuous work by industry forums and associations, including the Advanced Media Workflow Association (AMWA) and the Video Services Forum (VSF); alongside the Alliance for IP Media Solutions (AIMS) and in conjunction with the Joint Task Force on Networked Media (JT-NM), the industry is helping to shape the requirements and recommendations that can make the difference in flexible, reliable interoperability on a network-centric IP architecture.

Tasks essential to interoperability and sustainability (i.e., the ongoing management of systems at a software-defined level) are generally grouped under the heading of “network management.” IT professionals understand the basics of the processes and unfortunately often approach their needs and objectives through the grueling process of trial and error.

For many well-designed network centers, strict adherence to network management is what keeps the systems functioning. In systems where administrators have not been fortunate to have an overarching plan due to budgets, staff changes, inexperience or such, network management can be perplexing and is likely approached on an “as needed” adventure through the day-to-day operations of the entity.

In the upcoming paradigm shift into SVIP on Professional Media Networks (PMN), the trial-and-error approach will not work. As a primer to what network management involves, we’ll now look at the fundamentals of network management, sometimes referred to as network management systems (NMS), in IT vernacular.

Fig. 1: Processes and procedures typically employed in a network monitoring system (NMS), which are applicable in real-time studio video over IP applications and for storage management systems.

Network management consists of these factors and derivatives: configuration, fault, security, and performance, and accounting management (Fig. 1). The depth and practices involved in these network management tasks can be applied in differing ways depending upon the size or scale of the network involved, and the types of traffic that flow on the network.

CONFIGURATION AND FAULT MANAGEMENT
Knowing the impacts of varying versions (including updates) to software and hardware is a key objective in configuration and fault tolerant management. The processes involve monitoring the network and system configuration information so that the effects on operations within the network can be tracked and managed accordingly and consistently.

Important to these objectives is configuration file management—the verification that new config-files do not degrade the integrity of the network before an actual implementation takes place. Inventory and software management includes the discovery of all the network devices (a dynamic process that lists devices found on the network) and in some cases, an analysis of the software versions present (and past) so that regression testing, if necessary, can be administered without undo impacts on the network or its associated/connected peripherals.

Fig. 2: Steps in fault management and resolution. In Professional Media Networks (PMN), these steps may be provided on an audit network, which is specifically configured as a “watchdog” and “reporting” system that is designed to detect anomalies that humans and visual perceptions might not see or find.

In the PMN, fault detection (see Fig. 2 for fault management steps) may be captured directly by devices and reported to the controlling system. Trend analysis and awareness, along with integral audit networks (described later), can aid in the processes of fault detection, prevention or correction, especially in media-centric IP-video networks.

PERFORMANCE MANAGEMENT
Two areas that impact performance management are: (a) the monitoring, measuring and assessing, plus the reporting of those metrics established pursuant to the network design and functionality; and (b) the service level agreement in place when third-party providers are involved.

Storage management may use applications such as a MAM as the “orchestrator.” The process looks at how well activities such as file-transfers between storage tiers are working; migration to archive; backup systems; or delivery of files to the workstations.

The tasks are continually monitored in the background and are quite useful to the system administrator. As workflows and work processes change (an often-regular activity in content creation and delivery applications), the level of activities required by certain sections of the MAM and storage systems may be improved by adjusting parameters, workflow steps or the time of the tasks, accordingly. This is where performance management is valuable and functionality is often available in enterprise-level MAMs and storage platforms.

When services rely on third-party resources, such as network connections (WANs or MANs or when a service provider is engaged for other services, a written agreement between the provider and the customer—an SLA)—will contractually bind the expected performance level of network services. SLAs consist of agreed-upon metrics that should be realistic and measurable for each side of the contracted parties.

At a device level, SLA performance metrics may include CPU utilization, big buffer/medium buffer allocation, misses or hit ratios and memory allocation. Device-level performance statistics are critical to gauging and optimizing the performance of protocols that drive applications and computation power at higher levels.

For network routers, switches, aggregation devices and other components that support these various higher-layer protocols, performance statistics and technologies for intercity and intrafacility functionality should be monitored, collected and reported to gauge the effectiveness and efficiency of the network and then referenced to the SLA contract.

TUNING AND ANALYSIS
Compared to conventional IT or back-office data traffic, the flows on PMNs will increase significantly and be quite “dense”—that is, signals will be continuously running at or near the interface data link rate (e.g., 10 Gbps or 25 Gbps). Backbones connecting the aggregation points will run at rates in 40 Gbps and 100 Gbps profiles. Such activities will place a high demand on network resources.

For routine workflows and applications, the rates are unlikely to waver—that is, they will be consistent with the uncompressed video data rates for the applicable resolutions (i.e., multiples of 3 Gbps per 1920x1080p stream or dual-stream UHDTV/4K at 12 Gbps per path). PMNs, as a natural course, will be tuned (i.e., configured) for specific protocols and for prescribed data rates, buffer levels and essence flows. They will not be expecting variations in the streams unless specifically requested (re-subscribed) by the control systems.

Historically, network managers typically had only a limited view of the types of traffic running on their networks. In PMNs, where real-time services cannot be burdened by outside non-media-related influences—the network traffic will be confined to specific forms and structures.

Back-office traffic (email, office applications or social media) must be strictly forbidden on PMNs; in similar fashion to enterprise-class storage systems (using fibre channel or Gigabit Ethernet) where dedicated devices are optimized only for the storage data traffic specific to media files. Control and metadata for storage solutions are carried on a separate network, through separate switches and never cross into the storage data traffic itself.

Traffic profiling technologies provide detailed views of the flows in the network. In the IT world, two familiar technologies (RMON probes and NetFlow) enable the collection of these traffic profiles. In the PMN, this process will likely be vendor-specific and tailored to the exact forms of essence (video, audio and ancillary data) that the network will carry.

Accordingly, “big data” analysis techniques may well find their way into the next-generation PMN, providing new analysis insight into the real-time topologies.

SECURITY MANAGEMENT
Controlling access to network resources, according to provisioning guidelines established by the users or administrators, is the target goal in security management. Security involves regulation and monitoring of authorizations and authentications, and provides accounting (collecting and reporting) for billing, auditing or other designated purposes.

The PMN, like conventional data networks, cannot be subjected to the risks of intentional or unintentional sabotage. Processes in the security management subsystem monitor the normal routine operations (e.g., users logging onto a network resource, refusing access to those who enter inappropriate access codes).

PMNs will have to deal with influences such as USB files being added to a graphic system that then permeate through the PMN with unintended consequences, as well as unauthorized changes to the network topologies (e.g., the addition of an IP-camera, which isn’t automatically registered into the system).

Audit networks, intended to record and track activities, changes and even collect unrecognizable anomalies (e.g., minor network glitches, dropped packets, temporary and short-term oversubscriptions or excessive latency) may become a mainstream part of the PMN going forward.

Numerous other concerns and issues will become obvious as IP crosses the boundaries from file-based workflows into real-time dedicated (PMN) solutions. Users will soon need to develop sets of “best practices” and model configurations (topics beyond the level of this article at this time).

Needless to say, for PMN applications, users should leverage available manufacturer resources and/or consult with experts (e.g., systems integrators) who are experienced and familiar with all the aspects of security, control and IP implementation.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Latency is a continual concern, impacting nearly all forms of media and communication systems. It can be evident in an internet connection, a server, a disk system and in an IP network. We’re hearing a lot more about latency as we move from the SDI-world to real-time video over IP.

Latency is defined as the time interval (or period) between a stimulus and a reaction or response. Latency can be a “physical” property—as in the velocity of propagation (VoP) in a cable referenced to the frequency of the signal carried and is based upon the materials used in the fabrication of that cable.

Latency can also be a “signal transport” property, such as in the time of flight (ToF)—i.e., the methodology describing the time it takes an object, particle or acoustic, electromagnetic or other wave to travel a distance through a medium.

Fig. 1: Typical causes of latency due to propagation delay of the links (Dlink) and in processing delay at the nodes (Dnode) which must be accounted for in system design. ToF might include how the elements in a network (transmitters, receivers, switches and transducers) are engineered; while VoP is influenced, in part, by choice of cabling itself (e.g., Belden 1694A compared to 1855 for video or Cat5 vs Cat6a for IP).

Video latency—especially in a packetized IP-network—can affect reliability, stability and the ability to properly switch a line of video. Practices described in SMPTE RP168, the recommended practice defining the vertical interval switching point for video, help govern how real-time video-over-IP systems address the packetization of essence for live (real-time) video systems.

The needs for a live stream of video compared to feeding a monitor wall or encoder take on different perspectives, which must be carefully considered as signals negotiate segments of any network.

Latency in the recovery and reassembling of packets in a video signal transport network can affect the time it takes to reconstruct a complete frame (field) of video. Too much latency and you have a time inconsistency causing skipped frames or stuttering video.

Buffer capabilities also affect performance. If the network switch’s buffer is insufficient, the impact can result in an incomplete packet resolution, which could cause a series of video frames to be lost or a switch-routing or connection to fail—thus generating another improperly displayed image or a total loss of signal.

Performance and latency go together when looking at any system solution or application, which depends upon consistent and deterministic delivery of data. In today’s media implementations, audio and video essence (the data, sometimes called the payload) can take on variances depending upon where the data is presented in the workflow. One relevant and timely perspective (for the professional video industry) is embedded in the Real-time Transport Protocol (RTP) characteristics of networking; i.e., how the signals are impacted in a streaming-video vs. a file-based workflow.

Interested technical readers should read and understand such terminologies as RFC 3550 (the transport protocol for real-time applications) and RFC 4175 (the RTP payload format for uncompressed video)—both openly available and developed through the IETF (Internet Engineering Task Force).

SERVER PERFORMANCE

Servers bring with them their own set of latency and performance issues. Insufficient memory sizing; poorly managed caches or processor core(s) that cannot keep up with the signal throughput demands, which will result in a slowing of applications; signal throughput delays; or, in the worst case, computational errors, which hinder synchronization with the overall system’s demands.

Multicore servers available today have plenty of CPU power; yet getting that power to or from the network may be an issue. Network interface cards (NIC) and host bus adaptors (HBA) can be a server bottleneck if not properly configured, specified or implemented.

Traditionally these I/O components were locked to a single core processor. However, by using other resources, such as hypervisors and resource-side scaling (RSS), performance is increased by allowing the interface cards to distribute the I/O processing across multiple processor cores.

Another server virtualization technique deals with sorting the I/O tasks to the right virtual machine. This technique involves what is referred to as “virtual machine device queuing.” This VMDQ technology is productized by vendors who make I/O and processor cards. It allows the Ethernet adaptor to communicate with established hypervisor products to group packets per the best needs of the VM they are intended to be directed to.

STORAGE SYSTEMS

Storage solutions intended for media production have their own set of throughput, performance and latency issues, which are generated by varying factors. At the single disk root-level, and the most notable or easy to understand, are the impacts of “rotational latency” (sometimes called “rotational delay”), which is based in part on the rotational speed (RPM) of the drive.

This is a physical factor prevalent in every spinning magnetic disk or optical disc drive medium. Rotational latency is measured as the time that it takes the data on the drive platter in a sector to be positioned beneath the head for the read (or the write) process. The worst-case time-period number is found when the drive data has just passed the head, and the command to retrieve that data can only be met when the drive needs to complete a full revolution before being properly positioned to read that data.

Other factors include “seek time,” the time for the actuator arm to travel to the proper track on the drive; and “access time,” also called “response time,” the time it takes for a drive to transfer the data from the specified track to the drive electronics.

These factors are cumulative and affect performance, but in recent times we’ve seen additives to the storage mix that help alleviate performance-latency issues.

Solid-state devices, as complete replacements for HDDs or as supplemental caches for HDDs, help manage performance issues by (in the latter case) putting frequently accessed data into SSD caches. Here, routine calls for operational data or system metadata are stored in SSD devices, so accessing the HDD is reduced, improving efficiencies.

When editing applications know that they will frequently be playing a clip from the primary timeline, the well-designed app often instructs storage management to put that data into SSD. Timeline performance is improved and latency is decreased because the SSD has a significantly faster access time than does the HDD.

EVALUATING AND MATCHING

Storage systems should be demonstrated per the user’s specific implementations. For example, a NAS system designed specifically for media and entertainment will employ characteristics that can’t be met by other non-M&E storage solutions. One known evaluation test used in storage demos is to compare the load time of a set of files from a central storage array to the active NAS used in editorial or shared collaborative production.

Another is to check the ability to accurately scrub the video and audio in a high-resolution file (e.g., a 4K full-resolution sequence) and see how closely and smoothly the audio tracks the video.

Significant differences can be achieved by marrying the appropriate operating system (OS) to the file system, which traditionally have separate roles. When the file system is aware of the underlying disk structure, values are added.

In the past, file systems were traditionally created only on a single disk at a time. When two (non-RAIDed) disks were employed, two separate file systems would be created.

In a RAID configuration, this situation is avoided because the OS looks at only a single “logical disk,” which is likely comprised of many physical disks, on top of which the operating system places a file system.

In software RAID, the file system living on top of the RAID transform sees it is dealing with only a single drive. The implementation reduces read/write process complexities and hence improves performance. One pertinent example is ZFS (originally developed by Sun Microsystems, now Oracle), which is conceived of a combination volume-manager and file system. The concept allows the creation of many file systems that share a common pool of available storage. The storage system could be grown automatically because the file system is aware of the physical disk layout. This permits existing file systems to be grown automatically when additional disks are added to the pool. The new volume-space is then made available to every one of the file systems.

STORAGE IMPROVEMENTS

The evolution of storage “additives” is what storage vendors and solutions providers continually do to “build a better mouse trap,” as the colloquial saying goes. Users will inevitably keep changing their storage solutions because the improvements make consequential differences in their workflows. Pooling resources and using a storage manager product to relegate where that storage is used in the enterprise is just another way to extend the usefulness of legacy and in service storage.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Since storage has become a key component in nearly all media systems and workflows, one might wonder if “storage analytics”—the detailed information about individual and overall storage elements—are important to operations and management of the entire media production ecosystem.

Utilizing and understanding storage analytics is becoming more important as users configure virtual environments (such as VMware or Hyper-V) as well as conventional structured storage systems. When virtualized computer environments take shape across the enterprise, shared storage can also take on a much broader dimension. While multiple operating systems can operate from a single physical computer, how the information (data) generated in those environments is stored or managed becomes more difficult. Complex multi-element environments are never as easy to understand or manage compared to how a single OS, computer or server platform addresses a small NAS or group of drives.

Virtual machine environments and virtualization fabrics are subjects which have much depth, vary by manufacturer, and—for now—are beyond the scope of this month’s article. However, whether your system is VM-based or a mid-to-modest scale NAS or SAN, the value proposition in having analytical tool sets available that can manage the overall storage system is becoming a subject of interest to many IT managers and production workflow administrators.

If you’re fortunate enough to have a greenfield system with a full complement of new “raw” storage, then the options to configure and deploy this untapped storage pool are many. Yet, if your storage pool is a collection of many different sets of storage (as with many facilities) that are accumulated over various time periods, the efficient management of those storage “islands” takes on another dimension altogether. Often such systems may be comprised of DAS, NAS or SAN elements and will have little consistency in terms of volume sizes, I/O capabilities and distribution.

PERFECT CANDIDATE

The later (likely more typical) configuration is a perfect candidate for both storage virtualization and for a toolset capable of looking at the demands and stresses put onto the storage system. This process that uses these toolsets, in more modern terms, is known as “storage analytics.”

With the popularity continuing to grow in cloud technologies, a quick Google search for storage analytics will yield multiple sets of products provided by major cloud providers and by enterprise class storage vendors. Most products or services are prefaced by the cloud provider’s name (e.g., Azure) followed by the marketing term for the analytical toolset.

Some of the storage vendors will wrap the sentence to include VMware, Big Data, all Flash, scalable or some other storage marketing buzz word. The plethora of terms sometimes breeds confusion as to what you’re getting and how well it really does its job.

So, what’s under the hood? Why are storage analytics important? And what do they do or provide? How do they benefit the user?

STORAGE SYSTEM INSIGHT

Storage analytics are intended to provide insight into the physical and/or the virtual storage environment. Its tools should allow the user/administrator to optimize the storage system regardless of the complexity or the distribution. Additionally, there is also a business-sense component defined as “improving business performance” and “reducing costs.”

Fig. 1

A storage analysis tool set is designed to complement (i.e., improve upon) the traditional storage vendor-provided management systems. Analytic tools can help discover unclaimed usable storage across all the LUNs or volumes in a storage system, potentially providing significant cost-savings to any scale organization. When unclaimed storage is repurposed for other functions, users in turn may avoid having to increase storage capacity or might actually find performance is increased based upon what that orphaned storage is repurposed to do.

Fig. 1 illustrates an end-to-end data/storage analytics system, which consists of multiple probes within the storage and virtualization hardware (diagram’s left side) which collect the data via an FTP (or similar) server. The “Analytics Processor” core and database system (diagram’s right side) is the software-based solution that schedules and generates user reports based upon sampled information collected from the probed devices. The various processors in the core then create the trending, alarming and other data—reporting it to the Users’ fixed or mobile devices via web-based interfaces (concepts courtesy of Hitachi Data Systems’ data center monitoring solution). An optional BIRT interface (business intelligence report tool) may also be added, if desired.

VELOCITY, VARIETY AND CHURN

Trending analysis is another important tool found in most storage analytic toolsets. Key factors that may be helpful include the capability to look at data churn; that is, the “frequency” of change or the “retention time” of the data—information that can be used to adjust hierarchical storage management (HSM) policies or which application uses which data pool sector. As for the “velocity” of the data, is the system I/O optimized for the storage solution? Are there bottlenecks that hinder data transfers from one LUN (array or volume)?

Also in the toolset is a view into the “variety” of data—i.e., how much of the data is metadata, how much is full-resolution images (video, for example) and how much is proxy resolution (stills, short clips, thumbnails). Since storage should often be optimized for the type of data (small chunks, modest blocks or gigantic contiguous streams of files such as for 4K images), it is helpful to understand where those kinds of data are being placed, especially when you’re merging smaller volumes onto a larger storage pool.

TRENDING AND PERFORMANCE

Performance statistics and reporting are vital components in any suite of analytic tools. Many storage vendor tools provide basic, ad hoc reporting on storage performance. Few provide detailed or in-depth analysis with different levels of user-configurable granularity. The amount of detail available (i.e., the granularity of the information) should be adjustable based upon the system needs or the purpose of the investigation. The statistics used to generate the reports will then meld into the trending analysis programs, so the term of the trend and performance reporting needs to span multiple years.

At some point, storage will need to be either replaced or upgraded (expanded). Trending tools help predict where that point is so that the organization is not caught having to invest in more storage at an inappropriate time. The data points may also help determine when a cloud storage application is appropriate and to what volume (capacity) is proper.

BUSINESS INTELLIGENCEGATHERING

Ancillary products—those generally found useful in business intelligence reporting—are now being adopted throughout data centers and enterprise-wide large storage systems. One example, called Business Intelligence and Reporting Tools (BIRT), was developed as an open source software project aimed at providing reporting and intelligence via web applications created for rich client platforms (RCP). It integrates well with existing Java-based reporting applications. BIRT can be found integrated into some vendors’ storage analytical tool sets and by those who provide data center class storage as well as media-focused storage solutions. For storage analysis applications, BIRT is especially valuable if the organization is already employing BIRT methodologies throughout its other business units.

Intelligent analysis of your storage solution sets is valuable at any stage of the storage life cycle. At configuration, the tools help you analyze and optimize actual performance. At mid-term of the life cycle, the tools will provide benchmarks for peak performance, which can be used to validate or justify the storage solutions’ continued use or tweak its applications for improved optimization (should workflows change or be added).

Throughout the many storage work cycles, these tools help deliver consistent and concurrent reporting on the storage system’s status. And finally, the tools help to solve the most difficult of performance issues in a timely and convenient manner.

When selecting a storage solution for your organization, ask the vendor about the storage analytics tools available from the provider; then ask which third-party solutions they recommend or know have been added to their systems by other end users. The use of these storage analytic tools, from the start, may make a difference to your storage life cycle management two or more years down the road, and can be used to leverage change or improvements enterprise-wide.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage topics, read his book “Moving Media Storage Technologies.” Contact Karl at kpaulsen@diversifiedus.com.

Over the course of the past decade we’ve seen a proliferation of servers, often as commodity-based products from the usual sources, being added to facilities for nearly every system in the video production equipment room or the delivery network.

Servers—in a general sense—have steadily become the defacto device that provides the operational engine to much of the functionality seen in the modern digital media age.

From an implementation perspective, design practices have layered consecutive sets of hardware (e.g., servers and storage) to the mix of already a dozen to upwards of hundreds of other servers, each with a purpose aimed to support the next set of software applications or implementations.

Historically, each server would impart its own dedicated purpose within the workflow. Sometimes you would find the same functionality across different servers; see them in augmented segments of ingest, post production or news; and now we’re seeing them utilized extensively throughout the entire delivery chain.

With each device comes a primary connection and configuration to a network. Sometimes, depending upon the facility’s directives or budget, another secondary network interface to another network section would be added for protection or overall system resiliency. This continued duplicity and individuality of servers and fixed network topologies is costly and cumbersome to manage.

Fig. 1: Software-defined network (SDN) reference architecturePROFESSIONAL MEDIA NETWORKS

Virtualization is changing this legacy model. When similar functions can be distributed amongst a pool of servers whose functions can be modified to serve other purposes once a work task is completed, the efficiency of the system improves and the CapEx and OpEx costs are reduced. Virtualization, among other factors, is aimed at reducing the hardware components in a facility without reducing performance or other workflow support. Yet there is another evolving concept that has grown out of the compute (IT) industry and is filtering its way into professional media networks (PMN) for file-based and real-time video over IP.

This month’s introduction to SDN follows up from my April 2016 column on data- defined storage. In the IT industry, where entirely software-based systems are rapidly becoming the norm, the organizations that wish to accelerate application deployment and delivery—while dramatically reducing their overall IT costs—are transitioning to what is called “software defined networking” or SDN. Essentially, an SDN is policy-enabled workflows built on the principles of automation. In cloud-based systems, SDN technology is the catalyst that enables the cloud architectures, which provide for automated, on-demand application delivery, portability and mobility at scale.

DATA CENTERS AS CLOUDS

In a data center, the economies of scale are achieved in part through virtualization—that is, the sharing of like component to provide services as requested by the enabling software- based control and management systems. Referred to as “data center virtualization,” this functionality leverages SDN to provide for increased resource utilization and flexibility while in turn reducing the total costs for operations, including infrastructure costs and overhead.

Data centers can be considered clouds that may be private (i.e., services provided for internal organizations); public (i.e., any-services [XaaS] provided to others for a fee); or hybrid (where excess capabilities built for private services are sold publicly to entities outside of the internal organization). Regardless of the scale of the cloud data center, they all function primarily the same.

Typical cloud computing infrastructures— large sets of identical or very similar servers—live on large blobs (Binary Large OBjects), which form the common server structures. The blob is a collection of binary data stored as a single entity in a database management system. Blobs are usually composed of images, audio or other multimedia objects. Binary executable code may also be stored as a blob.

In a data center/cloud environment, myriad servers evolve much faster than in a traditional IT infrastructure built for specific functionality (such as for compute, back office or the like). Thus, blobs are not homologues (i.e., they may not be corresponding or similar in position, value, structure or function) to one another. In this scenario, server groups are fundamentally identical. For example, they are x64 architecture-based and they all share similar features, such as the peripheral component interconnect express (PCIe), serial advanced technology attachment (SATA) and Ethernet interconnects, which are essentially structured roughly the same way over repetitive subsystems.

This gives the data center/cloud the capability to scale—the capability to grow quickly without having to continually modify the physical (or software) infrastructures to “scale-up” to meet demands.

The management component for the orchestration of these capabilities is built on software-defined solutions including network components and virtual machines.

WORKFLOW DEFINED

An SDN allows systems to develop and adapt to varying workflows—that series of activities that are necessary to complete a task. Workflows are usually comprised of a series of stages or steps, each with a distinctive step before it (except for the first step), and are followed by another step after it.

Workflow steps are usually linear, but may also include looped steps with decision trees that allow the sequence to exit the loop and continue to the next step (or workflow) upon meeting certain conditions.

Workflows can be complicated and should be documented so they can be understood by external (human) sources or meshed with computer software built on a human-readable coding structure such as XML (eXtensible Markup Language). The written documentation is based on business processes modeled using principles found in business process management (BPM).

HYPER-CONVERGENCE AND ORCHESTRATION PLATFORMS

For a cloud or cloud-like implementation, the programming is called the “cloud orchestrator.” This platform manages the interconnections and interactions among cloud-based and on-premises business units. Orchestration may be applied to one or many sets of servers, but its best use is when applied to a system of servers, which, by themselves, have almost no understanding of what the adjacent server is to accomplish now or in the future.

Business objectives (e.g., a global ingest platform that brings content into a central repository) drive how the orchestration works and how workflows are developed (Fig. 1). SDN accomplishes these objectives by converging the management of network and application services into centralized, extensible orchestration platforms that automate provisioning and configuration of the complete infrastructure. Centralized policies collectively bring together disparate groups and workflows so they can deliver new applications and services in minutes, rather than the days or weeks required in legacy systems.

Depending upon its scale, this concept may also be known by the term “hyper-convergence”— a form of infrastructure with a software-centric architecture that tightly integrates storage, networking and virtualization resources (alongside other technologies) in a commodity hardware-based system; usually managed under some form of SDN.

SPEED, AGILITY AND FLEXIBILITY

When deploying new applications and business services, SDN enables the system to deliver speed and agility using existing components consisting of servers, storage and network switching. Programmability of these components is a key characteristic of a software-defining solution.

Another way to look at this is when various applications reside on servers that can be administered (delegated) when or as needed by the orchestration platform—while the associated server communications, instructions, process loading and data-steering throughout the network is being managed by the SDN.

SDN concepts are core components to making the emerging studio video over IP (SVIP) systems work using common Ethernet switches that can be configured per specific standards, some of which are currently in development.

Karl Paulsen, CPBE and SMPTE Fellow, is the CTO at Diversified. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Network attached storage, or NAS, continues to expand in acceptance and in capabilities. Where, for years, the storage area network (SAN) seemed to prevail for high-end, performance-tailored storage, NAS is changing those dimensions.

Using NAS as a storage solution is practical when the content includes sources such as surveillance or smart devices. As content types and storage dimensions evolve, users are finding they need to understand not only traditional NAS, DAS or SAN solutions; but they also must be looking at some recent trends in storage that are not so wellknown.

For those who have supported conventional NAS, they may now be looking toward “scale-out NAS” as its natural alternative. And for those who may have already outgrown the efficiencies of scale-out NAS, the latest revitalization comes in the form of “object storage.”

This month we’ll look at both of the later storage technologies, touching on why they each have applications based upon factors such as file sizes, quantities, search-ability and metadata.

SCALE-OUT NAS
Personal computers typically store data that is based upon a file system; a technology that can easily store information and organize data on a structured basis and which is human readable. The concepts in scale-out NAS let this data organization operate as one large global namespace that typically keeps the data stored in a hierarchical structure.

Hierarchical storage is best explained by suggesting that the file sets are structurally located within a system consisting of subfolders that nest among parent folders and reside on a common hard disk, solid-state drive or similar storage platform.

As storage requirements grow, the scale of the file system must increase. Scale-out NAS provides a solution that allows for storage capacity and file system scalability. The NAS device is effectively like a gigantic “C: drive” with capabilities to store millions (to billions) of files.

Fig. 1: a scale-out NAS architecture integrates with the traditional, and simpler, LAN-based client/server/storage topology. Early scale-out NAS systems were built on the principle of nodes (Fig, 1). Multiple nodes can be appended to the NAS (i.e., “scaled out”) so as to improve overall storage (i.e., “scalability”); yet often at the sacrifice of small-file performance. As the node count increases, overall storage system performance begins to slow. Metadata searches can be significantly impacted as the size of the NAS and number of nodes is increased.

Flash technologies have changed the small-file performance by a technique referred to as flash-first, a metadata management scheme that improves internodal networking speeds.

Today, the ability to store billions of files without diminishing overall performance, especially for metadata searches, is the norm for scale-out NAS.

Real-time data consistency is another key factor in scale-out NAS. When dozens to thousands of users must access the NAS, the ability to lock the metadata—in real time—is critical to preventing accidental overwriting or corruption while accessing the same files.

It is not uncommon for enterprise users to access the same files for different purposes. Such functionality mandates a metadata- locking system, which allows those files to be shuffled and/or their metadata renamed depending upon what the user’s purpose might be.

Human file management is another key factor for scale-out NAS. A typical use case example of both hierarchical and human-readable metadata is in the practical structuring of photographic images, which are often renamed, shuffled or moved into other folders for organizational or identification purposes.

The original image is likely known by the file name given to it during the capture process, such as “DSCN1234.” This is a rather useless name to anything other than a database manager. Users often will copy that file and rename it, e.g., “Johns 18th Birthday Present,” placing it (and several others) into a folder called “Family Birthday Pictures,” which resides inside another folder called “Birthdays.”

Scale-out NAS allows these file-naming techniques to extend throughout the individual’s library or in the case of the human resources department in an enterprise, across many interdepartmental file shares or home directories.

Other important factors for scale-out NAS include security management (user accessibility with read/write permissions); file-sharing among varying applications across a large NAS without the risk of one application overwriting the file in use by another application; and the ability for IT departments to have a central process by which all storage is globally controlled under one management system.

However, scale-out NAS is not without its limitations. Users are finding that even modern scale-out NAS solutions cannot keep up with the accelerating demands for storage.

We’ve often believed that unstructured data was dominated by content that was video-centric in nature. That case certainly holds merit, but as the Internet of Things (IoT) becomes more prominent, machine-generated data is gradually overtaking the amount of data generated by humans. Such emerging “web-scale” growth requirements are driving changes as to how files and file systems must be managed. Object storage is one of those solutions that simplifies the manageability equation.

Fig. 2: The familiar “rigid” hierarchical folder structure vs. the easier “everything in one bucket” approach for object storage that depends on containers instead of folders.OBJECT STORAGE
As a relatively recent concept, object storage is being applied to many storage provider solution sets aimed at addressing the limitations of scale-out NAS or other storage solutions.

Object storage mitigates the more complex hierarchical metadata attributes controlled through the legacy Portable Operating System Interface (POSIX) standard (Fig. 2).

Because object storage has only a few commands (e.g., Get, Put, Delete) it is an extremely simple-to-use interface. Its simple set of commands allows objects to exist with (globally) unique identifiers that are managed in as single “flat” address space.

The underlying principle in object storage is that data can be retrieved without having to know where or how that data is stored.

Object storage is governed in part by an extended metadata set, which is much deeper than that found in the conventional file system management tool set. Objects essentially become “self-describing” (the object knows what the information is about or what it is for). Objects can contain specific details about the application it serves, without limits as to what the content is, the size of the metadata contained in the object or how that information must be interpreted.

Objects therefore contain an extremely rich set of information, which users and programmers can leverage to allow their applications (and the storage that supports those apps) to perform much better. It includes attributes that allow for global distribution and infinite scalability. Its self-healing design yields high data reliability and provides bulk storage to be obtained at a much lower price point.

Users at both the enterprise level and the individual level are finding the feature sets in both scale-out NAS and object storage can overlap each other and, for many, yield new and important functionality heretofore unachievable with a single traditional storage solution approach. Object storage devices are now beginning to take advantage of NAS-like features as well, such as the expanding of interface-access methods across multiple applications, including both file and block storage.

No doubt, this will change how people buy on-premises storage and is, in part, how the cloud organizes its enormous amount of data on a much broader, global basis. Expect to see a lot of growth in these two storage technologies in the coming years.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage and media topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Tomorrow’s broadcast equipment rooms may look more like today’s data centers . As broadcast equipment technologies move closer to information technology-based systems, the way in which we design, build, monitor and account for operational costs is expected to change in near lock step with the technology we’re going to be building—or already are. With operational costs continuing to climb, items such as power, cooling, real estate footprints and even redundancy are likely to be watched more closely than ever in the past.

Many of the considerations for cost monitoring and control are somewhat obvious. For example, the overall power usage of each equipment rack determines the amount of cooling required; but that’s not the only factor in understanding cooling and the power that it takes to drive that cooling. As these spaces begin to look more like data centers than broadcast central equipment rooms (CER); the techniques used by data centers to control factors like Power Usage Effectiveness (PUE) become sectors we will continually need to manage.

VIRTUALIZATION ATTENTION

Efficient utilization of the myriad servers now being installed in the CER can have direct impacts on balancing the power usage and practicality of each system they support. This is why “virtualization” is getting more attention now than in the past. Many of the services used in the broadcast plant are seldom turned up to full capacity. Products that were once available only in a dedicated box are now being ported to a software-only domain where they can be loaded onto on-premises, common off-the-shelf (COTS) hardware—or even placed into the cloud.

Since the main point is to get the best use out of each device, there will eventually come a point where groups of servers, which have dedicated functions—but only run 25–35 percent of the time—will become consolidated and managed by systems that allocate the services on a needs-or-demand basis.

For example, if the functions of three servers are running at peak only 35 percent of the time, it is potentially possible that you can take those services and run them on a single server by throttling the workload and managing the utilization from a virtualized, pooled resource perspective.

UTILIZATION FACTORS

Media asset management systems and news editorial systems are notorious for throwing all sorts of servers into their workforce, but really not getting much more than 50-percent utilization during average work periods. Granted there are times when certain functions require those servers to run at nearly 100 percent; but that’s when the virtualization environment makes its best showing.

Assuming there are 30 servers in a broadcast facility (and that, frankly, is now a low number) for this example, picture the server mix as a group of 10 servers doing ingest, another 10 doing processing and the last 10 are parsed between output-conforming and shuffling media to/from storage. Ingest is likely to be unpredictable, so expect those 10 servers to be dedicated 80–90 percent. The processing servers are entirely workflow-governed. Processing probably gets the lowest utilization (or “loading”) percentage, say in the neighborhood of 20–25 percent.

Finally, those servers dedicated to do output-conforming and media shuffling aren’t much better at resource utilization than processors, so they run, for example, at only around 35–50-percent loading.

It doesn’t take a lot of rocket science to see that the idle time of better than two-thirds of the servers is greater than 50 percent. So why not consolidate the services to less servers and begin to cross utilize the hardware by finding software that can distribute the loading across only 20 servers?

If the services of processing (i.e., transcoding or proxy or quality control) and those used for output-conforming could run on the same servers, then the utilization efficiency of those 10 goes from 20–50 percent up to 55–75 percent. This leaves plenty of overhead to turn up one set of services during a peak time, or turn down other services and shift the application resources to other servers that are less loaded.

SPREADING EFFICIENCIES AROUND

If you could distribute this concept across even half of the servers in a modern IT-based facility, the utilization efficiency increases and the amount of wasted power consumed by the servers when doing “virtually nothing” goes way down. In the average data center, about 50 percent of the power consumed goes to the compute side (i.e., servers doing their job). Over one third of the power goes to cooling those servers with the remaining 17 percent of the power going to other items including lighting and losses due to electrical wiring (as “IR-loss”) and such.

Fig 1: Monitored power distribution unit (PDU) or “plug strip” with serial or Ethernet monitoring, which reports the voltage, current and power factor for each outlet; and the inlet power for utilization comparison and loading per device. What can be done to monitor and control these (in)efficiencies? One solution is to install PDUs (power distribution units, aka “cabinet distribution units”), in the data center, which can monitor each power outlet and each branch circuit (Fig. 1). This data is fed to a building management system (BMS), which can then keep track of each device in the CER by time and consumption in many dimensions.

In this model, it is also important to measure the current, voltage and power factor at the inlet side of each power strip using a technique known as “per inlet power sensing.” This lets the BMS understand power in versus power used.

PEAK LOAD OR LOW USAGE

Once monitoring is configured, each device can be tracked against time and workload. Since the power draw on a server is proportional to the amount of computing that occurs, a simple power monitoring application—typically found in a BMS—can then determine when peak loads or low utilization occurs. Over time, as opportunities come to transform the “single function per device” into a “shared resource,” there may be a point where shifting certain services to different times or reallocating the functions of a pool of servers to a virtualized set of services can change the power efficiency utilization curve.

If you live in an area where part of the commercial utility bill is based upon “changes” in demand, the leveling of the power usage across the most expensive power periods could make a difference in your overall utility bill since power loading and cooling could be spread more uniformly across the day or night. Reducing the changes in the power delivered can make quite a difference over the course of a billing period.

These concepts may not seem especially easy to understand or may not make much sense today, but over the long haul and as conditions change, the practicality in adding power and building monitoring capabilities could make a difference in the operational costs of the CER or data center. Most new “greenfield” facilities are already putting the extra monitoring capabilities in place today, knowing well in advance there may be secondary advantages that will come into play years later.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage and media topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Anyone with any type of high-performance storage system for a video playout server, play-to-air system or nonlinear editing solution of any scale has probably experienced this. One of their disk drives fails completely or you get that error warning of “imminent drive failure—change drive ooooX3Hd immediately!”

Some may procrastinate and risk certain impact. Others take heed and elect to change the failing drive, and still another group sits back knowing they’d planned ahead and bought an extra hot-spare and that the drive controller system will take over without human intervention.

Fig. 1: Characteristics of selected Redundant Array of Independent Disks (RAID) levelsRESILIENCY TO DATA LOSS
When a hard drive in RAID configuration fails there is a period whereby the system fault tolerance or resiliency to data loss decreases. Depending upon the RAID level employed or the protection scheme in place, the risks may range from moderate to serious.

The most significant and primary concern to the system is if another drive in the same array fails. Such a loss compromises the entire storage system and renders all the data in that LUN, array or possibly the system useless.

When the protective element in either the dedicated parity drive (as in RAID 3 or RAID 4) or the secondary protective parity set (as in dual-parity RAID 6) are no longer available, the period between then and when the array is rebuilt and back on line, at 100-percent service level, can be dicey at best.

When a failed or failing drive is detected, the administrator/maintenance technician must first replace the bad drive, which in turn triggers a process called the “RAID Rebuild.” RAID rebuilding is the data reconstruction process, which mathematically reconstructs all the data and its parity complement, so that full protection (with fault tolerance) is restored, in essence returning the system’s resiliency back to a “normal” state.

Sometimes when certain data checks or other errors are detected by the controller, the drive array may go into a reconfirmation period whereby checksums and/or parity algorithms perform a track-by-track, sector-by-sector, block-by-block analysis on each drive. Ultimately, parity or checksums are all compared and/or rewritten and the array is then requalified to a stable, active state.

STEPS TO RECOVERY
RAID fault tolerance involves a number of steps. In one example, should a disk fail, the RAID controller attempts to copy the resilient data to a spare drive while the failed one is replaced. Using parity data and RAID algorithms, which vary depending upon the RAID Level, parity data is then reassembled back to either the dedicated parity drive (as in RAID 3 or RAID 4) or is distributed across all the drives, as in RAID 5 or RAID 6 configurations. See Fig. 1 for selected RAID characteristics.

For other RAID levels, should one of the main data drives fail and no active hot-spare is available, a new HDD must then be installed. Then data from the other remaining drives is reconstructed using data extracted from a dedicated parity drive or from the parity blocks distributed across the array, back onto the new drive. Either way, the risks during the rebuild time are elevated until the new drive is brought online with all the reconstructed data and parity elements having been restored.

Large-scale drive arrays, those with hundreds of spindles (HDDs) usually have sufficient overhead, intelligence and processing bandwidth to compensate for certain fault issues. Those which employ intelligent RAID controllers can also be proactive. Should the RAID controller suspect or detect that a hard drive is about to fail, the proactive controller may begin the process of RAID rebuild to either a (hot) standby drive or signal the user to replace the failing drive, or add another drive in an available slot so that the RAID rebuild process can be kickstarted before an actual failure occurs.

FAILURES DURING THE REBUILD TIME
One of the drawbacks to RAID—when in failure mode or during a rebuild process—is that the performance of certain applications or processes may be impacted due to system latency. System throughput—otherwise known as bandwidth—may be reduced because: (a) not all the drives are functioning; and (b) the rebuild process takes away the I/O speed while it rapidly moves blocks of data sets from the remaining active drives onto the new/replacement drive.

The reduced performance can be especially noticeable when the array is relatively small, i.e., when the number of spindles (drives) is low; or when the individual HDDs are very large.

As hard disk storage capacities continue to increase, rebuild process times will take longer and longer. In some cases, for drives in excess of one to two terabytes, the rebuild process can last from several hours to several days. Of course, during this period, latency and the risk of another failure increases, resulting in performance and usability becoming further compromised. This is one reason, among others, that high-performance storage solutions tend to use smaller capacity HDDs (e.g., 300 GB to 750 GB) and may put many more drives into a single chassis or array.

These issues and considerations become part of the selection and decision process regarding how to choose a storage solution. For the more advanced video server or mediacentric products, those built for mission-critical operations, many manufacturers have already taken these conditions into account and provide sufficient fault tolerance or resiliency to “ride through” most of the more commonplace maintenance, upkeep and failure situations.

Another technology solution involves the use of flash memory or more appropriately, solid-state drives to either supplement the array (as a cache or secondary storage tier) or completely replace the hard disk drive altogether. I’ll explore that topic in greater depth at another time.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.