SAN JOSE, Calif.—Quantum has introduced the F1000, a new, lower-priced addition to its F-Series line of NVMe storage appliances.

Quantum’s F-Series was designed specifically for the video and image-based workloads, whether it’s HD video used for movie, TV and sports production, marketing and advertising content, or image-based workloads that require high speed processing, such as the data from a satellite feed, a drone, a car used in the development of new automated driver assistance systems (ADAS) and more.

Quantum’s customers can deploy the F-Series NVMe systems as part of the company’s StorNext scale-out file storage cluster and leverage the StorNext data management capabilities to move data between NVMe storage pools and other storage pools.

The F1000 is a 1U NVMe storage server optimized for performance, without the high-availability design of the F2000. Using a single-controller server and optimizing the F-Series software stack to run with less CPU, the F1000 offers the same streaming performance and response times as the F2000 at a lower entry price, the company said. The Quantum F1000 offers performance that is 5x to 10x faster than an equivalent SAS SSD storage array, at a similar price, according to Quantum.

The F1000 is available in two capacity points: 39 TB and 77 TB. It offers the same connectivity options as the F2000: 32 Gb fiber channel or iSER / RDMA using 100 Gb Ethernet, and is designed to be deployed as part of a StorNext scale out file storage cluster.

The F1000 leverages the same software-defined block storage platform that was introduced with the Quantum F2000.

A current trend that we’re starting to see become part of the architecture for possible shared storage solutions is NVMe (Non-Volatile Memory Express). NVMe is a new storage protocol designed to provide direct data transfer between central processing units and SSDs using a computer’s PCIe (Peripheral Component Interconnect Express) bus. It offers an alternative to SATA (Serial Advanced Technology Attachment) and SAS (Serial Attached SCSI) protocols and was designed to address the bottlenecks inherent in these previous technologies, unlocking the full potential of solid-state media. Its benefits include higher input/output operations per second (IOPs), gobsmacking throughput and greatly reduced latency. The specifications for these drives report them to be roughly 20 times faster than traditional spinning HDDs (Hard Disk Drives) are today.

Naturally, people are talking about NVMe, especially after IBC 2019 where Quantum and Western Digital, to name just two, showed off their solutions.

It’s the hot new thing. But do you really need it?

If your media creation environment needs extremely low latency and fast access to large amounts of data, then NVMe should certainly be considered. A couple of examples would be if you’re working in a collaborative environment with multiple streams of uncompressed or lightly compressed 8K or 4K video in use per workstation, or if you are orchestrating an event with a large number of concurrent ingest or playout feeds, such as an esports competition.

Additionally, VFX houses with large numbers of real-time or non-real-time renders might profit from every conceivable advantage to make shots available as fast as possible. There are some non-video workflows that also need extremely low latency and extremely fast performance as well. If I were trying to master global finances via high-frequency trading, build the perfect human via genomic research or profit from understanding the human condition via real-time big data analytics, you bet I’d want to build a fire breathing NVMe monster.

In our corner of the industry, unsurprisingly, we are seeing organizations dealing with large amounts of data as quickly as possible interested in NVMe, such as big media conglomerates ingesting a lot of high-resolution media and networks acquiring shows at the highest resolution so they can future-proof content. These organizations may still be delivering primarily in HD but they are archiving 4K files for a time in the future when viewers may expect higher resolution as a matter of course.

But outside this super high-end usage, most of our customers really don’t need a shared storage NVMe solution yet.

Most production and post environments currently don’t require extremely low latency or high IOPs because video playback is about large streams of sequential data. They’re ingesting and working with video on large, centralized, shared storage volumes that are using dozens if not hundreds of hard disk drives, which allows them to retain petabytes of information with great performance. Currently, that's still the best bang for the buck, and the best solution for most use cases right now because compared to a petabyte of HDDs, a petabyte of NVMe is exponentially more expensive.

That’s not to say that NVMe doesn’t have its place in your workflow today—it’s just a matter of the scale of adoption. People are moving towards solid-state technology wherever it is affordable to do so. For example, I wouldn’t buy a new workstation or laptop without one. On a smaller scale where NVMe’s reliability and performance truly shine, it can make all the difference in the world to a freelancer or editor working remotely from internal or direct-attached storage. In fact, desktop workstations and laptops equipped with NVMe storage outperform some of the SAN volumes we built for customers five years ago and certainly weigh hundreds of pounds less.

Additionally, a hybrid approach to shared storage—where some storage vendors provide a layer of NVMe cache on top of traditional HDDs—could be commonplace in the near future. This solution could provide the best of both worlds with the speed of NVMe drives and the capacity and price of HDDs, so long as the software controlling the data flows between the drives works transparently and seamlessly.

In conclusion, we believe that customers at the top end—like studios, networks and VFX houses—will use large NVMe shared storage volumes first. Additionally, the single, independent editor can make NVMe the cornerstone of their business by buying a machine equipped with a 1-4TB NVMe drive or attaching this storage to their machine via Thunderbolt 3. This option is cost-effective and offers 10 times the speed and performance of previous configurations.

Mid-tier media entities that comprise the majority of content producers and asset owners, who are currently working with compressed 4K or mezzanine HD video, might still consider the greater capacity and average performance provided by HDD storage solutions versus a smaller capacity NVMe storage solution with weapons-grade performance, as a better fiscal choice. You don’t need a supercar if a nice SUV will get the job done.

NVMe is in their future, too, as the industry inevitably moves to 4K, 8K, HDR and whatever other astounding and immersive new technologies are waiting in the wings ready to blow our collective minds. Right now, the decision to explore NVMe really boils down to IOPs and latency.

Ben Kilburg is the senior solutions architect, East, for Chesapeake Systems.

AMSTERDAM—Quantum is featuring the latest developments in its StorNext file system, F-Series NVMe storage arrays for video editing and rendering and R-Series removable, ruggedized storage systems for remote and mobile video during the ongoing IBC 2019 convention at the RAI Amsterdam.

The company is also making its executives available to discuss Quantum’s new line of Distributed Cloud Services and Cloud-Based Analytics Software, which allows users or Quantum’s own support team to manage and monitor environments around the world from one location.

“As the media industry produces more high-resolution, high frame-rate content, significant demands are being placed on the underlying storage infrastructure supporting these workflows,” said Quantum President and CEO Jamie Lerner.

“At IBC, visitors will see a more product- and technology-focused Quantum building on our leadership in both high-speed processing and long-term archiving of video and image content.”

Making its European debut is the Quantum F-Series ultra-fast, highly available NVMe storage array, which uses NVMe flash drives for reads and writes up to five times faster than traditional flash storage and networking systems. This performance supports the demands of real-time editing and rendering of 4K and 8K video, the company said.

The latest versions of the StorNext file system and appliances feature redesigned appliance hardware with two-times faster performance and other benefits, including editing and coloring of 8K content in real time, new predictive data movement and analytics capabilities, new cloud integration approaches and a simpler user experience, the company said.

More information is available on the Quantum website.

See Quantum at IBC 2019 stand 7.B07

HDD technology has long dominated because of good performance, relatively high capacity, and low cost per TB with flash. As price differences between HDDs and SSDs decreased, the desire for higher capacity flash SSDs has become worth the discrepancy – especially as the TCO can sometimes be lower because of density and power.

That’s where NVMe, Non-Volatile Memory express, comes into play. Designed to unlock the performance of all types of non-volatile memory, from flash SSDs to the latest persistent memory technology, this primer will describe how NVMe enables greater performance in both standalone and networked implementations, and some of the key use cases that can benefit from this speed.

Download the full White Paper

In the many changes in media occurring during this age of digital transformation, production facilities are facing decisions about whether to go IP, while trying to determine the impact of producing content in UHD-TV1 (4K) and above.

In January, the 2019 Consumer Electronics Show featured evidence that 4K is here, stable and “readily” available—while looking squarely at 8K (UHD-TV2) as the next great change in television displays of the future. Of course, with 8K comes a need for creating and delivering that content—and, as the cycle continues, a quantum shift in how to efficiently manage the changes required to the infrastructure.

Obviously, the volume of bits needed to produce all this new content won’t really get any smaller. To add to it, the need to produce good content for 8K likely means it is shot in high dynamic range and at a minimum of 4K in resolution to realize the value of the larger screens, which will depend upon upscalers for quite some time.

Fig. 1 shows just how image sizes will relate to increases in bit volumes in order to meet the requirements to deliver high-quality video in the future.

MULTIDIRECTIONAL SCALABILITY

Throughout this past decade, applications associated with SaaS, AI, VR, social media and image resolution have demanded that systems scale in multiple dimensions. Peak demands continually change, forcing previously unseen growth in data sets and its management. New frameworks are needed to address these new data-intensive workloads in proportions that legacy solution sets cannot meet.

System resources are now being decoupled, allowing them to be scaled independently and turned into services versus how they were treated heretofore. Those who relied on the “shared storage” model are finding there is insufficient I/O (input-output) performance and excessive latency (i.e., throughput boundaries) to meet new demands, which goes for network interfaces and servers as well as storage.

Flash-based designs, now approaching more than 20 years since their introduction, are reaching a peak in terms of serial data I/O and transfers. No longer is it efficient to simply replace a spinning hard drive with an SSD and then tweak in performance on a legacy SAS or SATA interface.

Today, the latest up-and-coming technology is Non-Volatile Memory express (NVMe)—a scalable host controller interface coupled to a storage protocol that accelerates data transfers between client and/or enterprise systems that utilize high-speed PCIe (Peripheral Component Interconnect Express) based solid state drives (SSDs).

COMMANDS AND QUEUES

At least two primary factors affect SSD performance—commands and queues. A traditional SATA device will typically support up to 32 commands in a single queue; with the SAS device supporting up to 256 commands in a single queue. Here is briefly how these two constructs work:

Based upon the anticipated workload and system configuration, host software will create “queues” (positions or slots of availability)—up to the maximum supported by a controller. Often this is regulated by the core processor and is limited in number to avoid locking and to ensure the data structures are created per the cache of the core’s processor(s) without encumbrances.

A circular buffer, known as a Submission Queue (SQ) with a fixed slot size is used by the host to submit “commands” for execution by the controller. Each SQ entry is a command. A Completion Queue (CQ) is another circular buffer with a fixed slot size that is used to post the status for completed commands. The number of queues varies by the application—enterprise applications range from 16 to 128, with client queues only between 2 to 8. Block sizes for both are 4 KB (and above with NVMe protocols).

LANE CHANGES AND GIGATRANSFERS

Understanding the true meaning of the PCIe solution set can be complex, given the variety of interface opportunities and the interdependency of the I/O to and from the device being attached to the bus. Calculating PCIe bandwidth can be even more challenging, especially when giving rise to new terms such as Gigatransfers (GT/s) that are interchanged with rates and speeds, such as gigahertz (GHz). Rather than look at the explicit details of each “PCIe generation”—we’ll provide some introductory information about the evolution of PCIe in a simpler perspective.

Platforms utilizing PCIe connectivity have continued to rise, moving from Gen1 (24 lanes) to Gen2 (36 lanes with a doubling of bandwidth), to an I/O performance of 1GBps per lane in Gen3.

A lane is a data-transmission link, which consists of two pairs of wires—one pair for transmitting and one pair for receiving. Consumer PCIe slots can be 1, 4, 8 or 16 lanes. Packets of data move across the lane at a rate of 1 bit per cycle. The 1x link (one lane) carries 1 bit per cycle in each direction (hence two wires per direction times 2). A 2x link (two lanes) utilizes eight wires and transmits 2 bits at once (per cycle) in each direction. The numbers grow with successive PCIe generations.

PCIe Gen1.x and 2.x use 8b/10b encoding, which results in a 20% performance overhead. The encoding converts an 8-bit data set to a 10-bit character set, thus a per-lane 250 MBps bandwidth can only carry 200 MBps. When utilizing the gigatransfers parameter from the gigabytes bandwidth number, the numbers have a non-uniform change that is best presented in a table, which reflects the encoding, transfer and speeds for comparison (Fig. 2).

SSDs supporting Gen2 with eight lanes, deliver over 3 GBps; doubling that on Gen3 interfaces to 6 GBps for a single device. Encoding is now 128b/130b, resulting in only 1.5% overhead. Latency is also reduced and the ability to directly attach to the chipset or to a CPU was also revealed.

IT DOESN’T STOP THERE

AMD announced at the 2019 CES it would be the first to support PCIe 4.0 at either the enterprise or desktop (client) levels. Ironically, in May of this year, the PCIe 5.0 specification (providing four times more bandwidth than PCIe 3.0) was announced even before Gen4 (PCIe 4.0) was shipped.

Just how far will this go? Why does the development continue when, for example, Gen5 essentially has the bandwidth of a 100 GbE (Gigabit Ethernet) connection—equivalent of about 63 GB per second at 16x speed? Certainly, home users don’t need these values and even enterprise users might question this proposition—especially given the limits of the SSDs they would connect to and the costs of the switch gear—irrespective of the continually decreasing costs.

One evolving scenario for these exponential data rate growths is that of the “always-on” (vs. the “always-connected”) compute platform level, which is taking shape with the always-on model seeming to lead the race because of battery life expectations. For the media and entertainment industry, i.e., those routinely creating UHD/4K content, storage refreshes can now take less space, require less power and cooling, and increase performance multifold over previous storage solution sets.

EMERGING PROTOCOL

Current storage system advances are now based upon the latest NVMe protocol. Utilizing NVMe with multicore processors aids in removing bottlenecks that conventional interfaces alone experience. NVMe brings about highly scalable new capabilities for accessing storage media at high speeds, which in turn is enabling more growth for data-driven marketplaces including media, video production and post.

When you’re in the refresh mode or considering moving to UHD production, take a look at storage solutions incorporating NVMe, especially those who offer the repurposing of existing SSD, which you may already own.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Quantum has launched the F-Series, a new line of NVMe storage arrays “designed for performance, availability and reliability.”

Non-volatile memory express (NVMe) flash drives allow for massive parallel processing, while the latest Remote Direct Memory Access (RDMA) networking technology provides direct access between workstations and the NVMe storage devices.

[Webinar: Improving Post-Production By Better Utilization Of NVMe Enhanced Storage Resources]

These hardware features are combined with the new Quantum Cloud Storage Platform and the StorNext file system, resulting in powerful end-to-end storage capabilities for post production houses, broadcasters and other rich media environments.

The F2000 is a 2U, dual node server with two hot-swappable compute canisters and up to 24 dual-ported NVMe drives. Each compute canister can access all 24 NVMe drives.

Quantum’s Cloud Storage Platform is integrated with StorNext, allowing customers to reduce total cost of ownership in areas including post production, sport video and rendering/simulation.

Quantum F-Series will enable customers to:

• Ingest, edit and finish content with lightning fast performance – the F-Series is up to five times faster than traditional flash storage/networking, delivering extremely low-latency and hundreds of thousands of IOPs per chassis
• Gain predictable, low-latency access via fibre channel or ethernet – users can reduce infrastructure costs by moving from fibre channel to ethernet IP-based infrastructures
• Meet performance requirements with less rack space – customers who use a large number of HDDs or SSDs to meet their performance requirements can gain back racks of data centre space

Jamie Lerner, Quantum president and CEO, said: “This is the most significant product launch we’ve done in years, not only because our F-Series product line will enable our customers in media to produce more great hi-def content faster over IP networks, but because the F-Series is the first product line based on the Quantum Cloud Storage Platform.

“This platform is a stepping stone for us, and for our customers, to move to a more software-defined, hyperconverged architecture, and is at the core of additional products we will be introducing later this year.”

Quantum will showcase the F2000 at its booth SL4409 at the 2019 NAB Show, April 8-11.

Flash memory provides a needed level of storage performance for media and entertainment applications which now command, for UHD and beyond, massive amounts of storage and speed to meet the growing amounts of content being generated. New high-res applications demand much higher storage performance than conventional enterprise back office applications. Flash memory is helping enable that performance—but with it, the physical media also commands interfaces that provide the best value for the applications and its associated storage.

Non-volatile memory express (NVMe) is a trending technology that utilizes Flash storage more effectively and efficiently. The harmony of solid state and rotating storage media will continue for the near term, and for the undefined future. These respective storage mediums will endure, providing complimentary values for ambitions such as more storage, better storage, and faster throughput with a reduced hardware footprint, at less cost.

[Read: A Solid State of Non-Volatile Memory]

MORE THAN A 2X BUMP

At the previous two Flash Memory Summits (2016-17) both NVMe and the PCIe 4.0 bus were hot topics. Yet right alongside the 2016 Flash promotors, the SCSI Trade Association reminded the industry that “a new serial-attached SCSI (SAS) technology was on the way.” That new technology took the SAS ecosystem from the 12G level to a usable 24G SAS (24GBps, serial attached SCSI). Promoters unveiled this advancement as more than just a “two times bump in speed over the previous data rates for 12G SAS” (refer to Fig. 1 for the SAS technology roadmap).

The interface of choice for mission-critical storage applications remains SAS; moving the interface from 12GB to 24GB yielded a major refurbishment in the technology. Updates included more efficient 128b/150b encoding, with SAS Protocol Layer (SPL) packets and Forward Error Correction (FEC). The transmission signaling rate is specified at 2.4GBbs (i.e., 22.5 gigabaud rate) which still retains compatibility with earlier 6G and 12G solutions.

Changes which help achieve this faster throughput include an FEC field, 20 bits in the SPL packet that aid in error detection and recovery. An SPL packet is a 150-bit block that includes a 2-bit header and a 128-bit packet payload, plus the FEC bits. At the deep-dive level, additional features added include changes in: binary primitives (in the SAS Link Layer); primitive parameters; serial management protocol (SMP); open priority; and inter-expander fairness arbitration enhancements. Details can be found in a technical overview document for Serial Attached SCSI, which is roughly 1,000 pages and whose details are beyond the scope of this overview.

LOWERING LATENCY

Regardless of the application for media and entertainment or others such as transactional trading on the stock exchange, latency has a tremendous impact on storage performance. Recalling from our previous article—NVMe is an industry standard that is optimized with a new storage stack featuring a low latency, efficient and scalable protocol streamlined with a revised drive command set that uses fewer clock cycles per IO operation.

Comparing spinning magnetic drive latencies for hard disk drives (HDD) to those of Flash SSDs, conventional Flash NAND technology offers a 100x reduction in latency over HSDs.

Latency figures decrease further when NVMe eliminates the 20 microseconds of latency found in the SSD NAND (whether SAS or SATA) implementation. “Next Gen NVMe” will now drive NVMe to deliver “4KB operations in under 10 microseconds,” according to a presentation at the 2016 Flash Memory Summit.

CHANGING THE MESSAGING

What do these memory improvements provide to the media and entertainment (M&E) industry? For starters, it enables audio/video to become the dominant medium for the carriage of information and content. In a February “New York Times” text message to mobile subscribers we saw, “What you are doing now (i.e., ‘reading text on a screen’) is likely going out of style.” The Times’ text infers that A/V will replace the text messaging we do today and in the not-too-distant future.

One of the tasks necessary to achieve this change will be to increase the speed of the delivery while at the same time improving the processing and memory requirements of the channels which deliver the information. The new 5G networks may help this, but more is needed. This same prolog can be applied to production and post-production workflows for M&E.

[Read: Non-Volatile Memory Grows In Popularity]

We have already seen the impacts of 4K over HD in terms of resolution, quality and image perception. While well-produced 1080p content can be wonderfully upscaled for 4K displays; in many cases there can be very satisfying results when the content is originated as 4K and then downconverted to 1080p and displayed on a high-end HD (1080p60) television system.

The real impact for UHD/4K, once available in more delivery systems, will be in the use of higher frame rates (1/120 second frames instead of 1/60 second) especially for sports. Add the higher resolution per frame and a higher density of pixels per unit screen area, and it could make today’s conventional HD images look like older analog 1-inch videotape, comparatively.

IT TAKS A LOT OF MEMORY

To reach these goals, it takes memory, a lot of it; and fast memory coupled with much higher bandwidth (selected video formats described in Fig. 1). Compression helps to a degree, but with the improvements in compute and networking technologies, more productions are switching to full-bandwidth, uncompressed video for their editing and post-production workflows.

Digital storage for media workflows will continue to advance in these areas. Flash, coupled into NVMe interfaces, make these higher capacity, faster and better memory solutions possible. We’ve seen Seagate produce a 12TB “BarraCuda Pro” 3.5-inch HDD with SSD-like performance, and SanDisk’s “Extreme PRO” CFast 2.0 solution in a 256GB form factor capable of 525MBps read (450 MBps write). These devices are the tip of the iceberg, as the products become the workstation’s local drives (i.e., the Seagate HDD) and the camera capture’s memory solution (i.e., the SanDisk memory)—key components needed to work in higher resolution and higher frame rate production workflows.

ACHIEVING THE TARGET

Along with this perspective on higher resolution, higher bandwidth production—another element which will consume massive amounts of memory is video on demand. According to Coughlin Associates, the shipping capacity for VOD storage will increase 8-times, from the 2,500,000 TB (terabytes) in 2016 to a whopping 20,000,000 TB (per year) in 2021. Add the other data sets, which aren’t getting any smaller, and the requirements necessary to fulfill all these ambitions is quite an undertaking. When AI and VR take hold, cloud services may not be enough to satisfy such exponential changes in storage volumes—let alone the means to transmit (wirelessly) the demands to the end users.

To manage the physical maintenance issues of storage using HDDs could require small armies of support teams (quite possibly robotic) just to change the drives in even a modest data center. Flash memory would need less support, with the R&R (rescue and recovery) cycles for SSD upkeep probably hundreds of times less. Technological changes would likely outpace the life of the SSDs in service, making storage a disposable commodity.

This is the future for storage. Mechanical drives, besides reaching physical capacity thresholds that may limit their overall performance, are probably in their last decade of useful life—at least for large scale M&E solutions. Single unit workstations may continue with HDDs, where cost-to-capacity requirements are different than in datacenters—although the migration to all SSDs is very much apparent in tablets and mobile devices. Watch for some dramatic changes in storage and Flash as more video enters into our lives and better images become an expected case and not a “luxury once in a while” alternative.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.