The term “cloud” originated from diagrams of large-scale data networks and, for a long time, the cloud was just a “means to transfer information from one place to another using someone else’s computer.” In time, privately controlled services emerged that allowed users to store data with “cloud service providers,” new businesses that permitted access to their “private” data from anywhere.

In truth, the user’s data lived on computer systems owned and hosted by a cloud service provider. Those physical places became known as data centers, consisting of racks and racks of computers, storage devices and networking devices that communicate and/or support many compute functions. Such services must be capable of expanding—as necessary—to meet the user demands for short-term, long-term and immediately accessible data housed in locations typically unknown to those end users.

Private Clouds
Large companies often have their own “scaled” data centers located at their physical buildings, which allow for personalized management and cost a great deal to fabricate, support and manage.

Today, there are two general meanings of this “data center” terminology:

• It’s where the term “on-prem”—meaning “on the premises” comes from—inferring that self-managed compute services (usually servers and computer elements), storage and input/output network management are physically placed in the company’s owned-and-operated facility (such as a hardened warehouse full of electronics, cooling, backup power and security).
• You can host your data in your own datacenter “on prem,” or you can host it with another service “as in the cloud.” Private companies are now building entire data centers to either outsource/lease space (only) or house entire systems solutions for users to place their own gear into or rent the service provider’s systems on a “square-foot” basis or a “rack-by-rack” space basis—with or without maintenance or support by a third-party organization.

Cloud Structure
A cloud can be considered a business when it is owned and operated by a “recognized” entity such as Amazon (Amazon Web Services), Microsoft (Azure), Google, etc. But they may not be the only “cloud resources”—companies that offer software services under their own “private” cloud (e.g., Wasabi or Comcast). Such cloud organizations or structures may also provide intercloud offerings, allowing them to scale across larger data sets, bridge various specialized “data centers” or even sublet entire facilities to a particular single entity as needed.

For on-prem solutions or even large-scale public data centers, monitoring platforms are “managed over-the-top subsystems” that may drive an entire solution set platform (data center) or link groups of data centers. For an on-prem environment, such a platform could require a large up-front commitment by the organization.

If your organization is employing a cloud-based DevOps solution with rapid iteration and continued live or real-time practices that involve continuous results reporting and direct-to-developer feedback, though, it may be difficult to predict or understand the depth of commitment necessary on the front end, let alone what might occur as the systems scale upward in response to growing needs for client services or compute and storage expandability.

Self-deployed monitoring systems may also generate a lot of unused capacity and wasted resources. That is unpredictable as the software-solutions processes move through the various “bring-to-market” stages. Even when the DevOps solution is built for internal or local operations only, the self-built/self-managed monitoring platform can overwhelm the IT or engineering services with unpredictable consequences.

Today, there are many potential open-source monitoring solutions available to organizations; some strictly on-prem and others entirely cloud-based. Any improperly integrated solution could generate a lot of unused capacity and wasted resources—as reported by service providers, vendors and end users.

That’s not to mean an experienced organization familiar with today’s most relevant trends (and with sufficient cloud management solutions experts in-house) should not or cannot develop a comprehensive, scalable monitoring solution that fits its needs. However, the risks and costs can escalate as the environment expands or as the scale mushrooms. It is a complex balancing act that can make or break the organization’s performance or profitability.

Before venturing into this domain, be sure to understand the overall solutions development processes—especially the real-time management core initiatives and the harmonization of the operation’s IT functions.

First, Defining DevOps
DevOps is a collaborative approach that merges software development and IT operations—and is nearly always a live or real-time environment. DevOps typically combines people, processes and automated tools to build, test and release software much faster and with greater reliability.

Throughout the DevOps process, software is constantly monitored while in development or in use. During the testing and “spinning up” process, user issues generate large amounts of data and feedback that go directly back to developers to be quickly improved. Once deployed, the procedures usually continue and may require more capabilities than an “on-prem” solution might be able to handle.

In the cloud, DevOps may function by using “on-demand” cloud infrastructure (Fig. 1) to automate software delivery, manage infrastructure through code and auto-scale resources dynamically to meet delivery needs, adjusting the flows by reacting to anomalies, including failures, crashes or data overruns. In the cloud, instead of manually configuring physical servers, teams write code to provision environments, test automatically and deploy updates while continuously monitoring application health.

Infrastructure as Code (IaC) is the process by which operations teams define network servers, databases and environments using configuration files (such as Terraform or AWS CloudFormation) rather than manual clicks in a portal.

Continuous Integration and Continuous Delivery (CI/CD) is an accompanying cloud-native service (such as AWS CodePipeline or GitHub Actions) that permits the automatic testing of code as soon as developers commit it. Once verified, the cloud platform automatically pushes the software update into production.

Data Centers
One of the “DIY” challenges is when an organization is driven to write all of the integrations itself and then forced to manage those integrations long-term. Novel “open-source solutions” require a great deal of upkeep, attention and maintenance.

Users report they almost need to have a Ph.D. to set up effective and sufficient monitoring capabilities that can address the ever-growing needs of a DevOps environment.

Some (but certainly not all) system solutions vendors may either make it difficult to figure things out intentionally or require long-term solutions support contracts, resorting to extensive Googling or digging into all kinds of online forums for answers. Much of that information will likely be inapplicable to the organization’s actual needs without considerable DevOps adoption (while under live operations).

Monitoring for Success
Effective end-to-end monitoring is crucial for enterprise DevOps teams to ensure high-quality, scalable and secure software delivery across complex environments.

Key requirements for enterprise DevOps monitoring platforms (Fig. 2) must be properly selected to ensure scalable, reliable and efficient software development and operations. The cloud on its own is not generally designed to develop those platforms without a considerable amount of code or structured elements that are specifically fashioned to the needs of the user’s organization.

Effective end-to-end monitoring is crucial for enterprise DevOps teams to ensure high-quality, scalable, and secure software delivery across complex environments.

In our next installment, we will dig deeper into issues on dynamic scaling, integration and vendor support, the importance of documentation, how to establish real-time notification and developer access without infrastructure exposure and end-to-end data capture across the DevOps lifecycle.

Let’s review some of the important feature sets typically found in a cloud solutions provider.

First, the cloud provider should almost always store or process your data in multiple locations, aka data centers. These data centers provide the physical elements for connecting all of your data, anywhere. Data access will generally include cloud apps, databases and hundreds to thousands of both on-prem and off-prem systems, using “prebuilt” connectors that integrate the solutions handling your data and allow it to be processed through established services.

A cloud provider should be able to effectively leverage your existing infrastructure with an ability to query or analyze your data with features including replication, movement/migration and “rework.”

‘AI-Ready’ Data
Given the global emphasis on artificial intelligence, one would almost expect this service-level statement—“all our data is AI-ready”—given the levels of artificial intelligence that the marketplace continually promotes, irrespective of the reference or workplace. Fig. 1 depicts a workflow inside a cloud that could aid in preparing data for AI-ready states or actions—ideas shown in Fig. 2 generally feed back into systems, as shown in Fig. 1.

AI-ready data means that your information has been systematically prepared, evaluated, managed and governed to meet the needs of AI projects. With financial-related data, expectations are that transaction records are properly prepared before that data is fed into an AI model.

Assume certain checks that your (cloud) services provider can include or package can identify patterns (or repetitive series of characters that could flag harmful routines that might represent fraudulent transactions, loops or means to generate a code sequence that would alter, falsify or get a back door to an unwanted action).

In retail applications, your cloud provider might offer “AI prep” capabilities and readiness for applications such as “demand forecasting,” which uses historical data on sales volumes and costs, as well as comparative product details that can be shared across hybrid and multiple cloud providers located regionally, globally or both.

For organizations (like original equipment manufacturers) storing preventative or predictive maintenance for industrial purposes such as aircraft maintenance, the cloud services provider should be capable of tracking and cataloging short-term and long-term historical data, plus real-time data derived from sensors and performance variables. Applications for the cloud-storage systems would leverage and train AI models to accurately predict equipment repair times, schedules and relative downtime.

Sometimes referred to as a “digital vault,” immutable storage is a paradigm where information, once written, cannot be modified, overwritten or deleted for a specified retention period. It is also referred to as WORM (write-once, read-many) storage or object-locked storage. The opposite term is “mutable storage,” which can be edited, replaced, modified or destroyed at any time.

Unlike mutable values, an immutable value or content cannot be changed without creating an entirely new value. For example, in JavaScript, primitive values are immutable—once a primitive value is created, it cannot be changed, although the variable it holds may be reassigned to another value.

Supply-Chain Security
In an “open source” age, malicious activities are common and almost expected in nearly every software and data system—and especially in cloud services. In e-commerce services (such as eBay or Etsy), users place assurance expectations on their vendor’s services, who in turn rely on the respective e-commerce company to “pre-protect” the data and services of their customers and clients, using industry best practices and some of the services listed in the following:

• For an in-depth understanding of how certain software is protected, Software Composition Analysis (SCA) emphasizes control over inventory, dependency mapping via Common Vulnerabilities and Exposure (CVE) and license tracking, as well as enforcement policies in pull requests (PR) and continuous integration (CI) before release.

Note that SCA also stands for Strong Customer Authentication, a regulatory requirement under the European Union’s Revised Payment Services Directive (PSD2), designed to reduce fraud in online payments. Strong Customer Authentication requires at least two of three elements—knowledge (password), possession (phone) or inherence (fingerprint)—for payment validation.

• CVE is a standardized, international dictionary of publicly known cybersecurity vulnerabilities in software and hardware. Managed by the MITRE Corp. with U.S. government support, it provides a unique ID (e.g., CVE-2024-1234) for tracking flaws. It facilitates fast, secure communication about threats and feeds the National Vulnerability Database. There are currently over 330,000 CVE Records accessible via download or keyword search.

Securing Against Ransomware
A “zero-trust” architecture does not implicitly mean “don’t trust anything,” but it does signify an architecture that is harder to breach and is an upgrade to your access control and much more. Zero-trust often demands multifactor authentication at all access points and insists that all connected devices are regularly updated and well-maintained.

Hybrid Cloud Vulnerability
In today’s hybrid cloud world, enterprises struggle to keep track of the slew of certificates managed by different siloed teams and tools. The lack of a centralized view of health increases the risk of application disruptions due to expired certificates. In the AI and open-source era, vulnerabilities in open-source dependencies expose applications leading to unwanted attacks.

Ignoring production usage of open-source packages can lead to breaches and disruptions. Malicious bad guys often weaponize disclosed vulnerabilities quickly, shrinking your remediation window at each cloud source transition (e.g., in hybrid or multi­cloud). You’ll need regular, thorough monitoring to be sure your access control is tight. And you must improve management by limiting access to individual components in the network.

A Flexible and Forward-Thinking Approach
There’s segmentation, and then there’s ZTS (“Zero Trust Segmentation”). You can be certain of some things—the big ones include:

• Cyberattacks are unavoidable: Statistics show this to be true, yet for many organizations there’s a surprising lack of preparedness.
• Cybersecurity mindsets are often outdated: Even with continued investment in perimeter controls, organizations still get breached. When you recognize and accept that breaches are inevitable and start to assume breach, you can focus on isolating them and stopping their spread. ZTS is by far the fastest and easiest way to do that.

ZTS is a flexible and forward-thinking approach that is “AE strengthened” by default. “AE strengthened” refers to key applications, including structural health monitoring using Acoustic Emission (AE) monitoring or, contextually, the bolstering of organizational or technical capabilities (e.g., AE engineer, Advanced Energy—refer to Fig. 2 for example details).

Who’s Responsible?
Essentially, it is the duty of the cloud service provider and end user management to ensure appropriate safety factors are in place and routinely updated before opening the door to widespread public use of cloud-service capabilities. In a future discussion, we’ll look at cloud egress fees and egress payments, an area that’s becoming a bigger part of modern cloud operations.

In any business’ financial world, the company officers, sales leaders, engineers and such are often asked to justify not just the “reasons” for making decisions, but are also to validate the return (usually monetarily) on what assumptions, decision or investments (i.e., the “costs”) are involved in making that decision. Besides just the project “budget,” this summary is often known as the “return on investment” (aka “ROI”)—and this generally becomes a determining factor in making a “go for it” (or not) on the project or its expenditures.

Those metrics are needed to assess the “investment” in terms of expenditures, i.e., the dollars for capital or operating, the number or size or resources (people, space hardware or even outside services) and finally “how long will it take to recover or begin seeing meaningful returns” (profits). Figures or merit may further involve what will be the volume (size) of those returns in terms of new (or reduced) people, expected costs, efficiencies and performance.

Given the astounding references to AI given in everything today…a great many will emphasize these “ROI factors” while at the same time attempting to understand what is gained by going “down the AI path” and the change requirements needed to realize this ROI when employing AI in the solution. These factors can be very different elements which are applied depending on where, how and by what means the adaptation of AI concepts will be utilized…especially given the acceleration in tech, media, engineering, manufacturing, etc.

To examine the impacts of AI, many find that using “the marketing segments” is a worthy workplace factor when deciding what elements of AI are best applied and where as the business evolves. We’ll use the “marketing segment” as the strategy for example in this article on AI in the marketing era.

AI in the Market
A recent marketing-focused white paper from iterable.com stated that 47% of marketers are drawn to AI for its ability to make their work more efficient. Higher efficiency means more time to strategize on how to reach customers in a meaningful way. However, the way marketers view AI goes way beyond that.

A white paper in “Ad Age” highlighted some of the findings from a Wakefield Research survey of 1,200 marketers worldwide. The survey took a deep dive into how companies are taking a second look at AI and includes many of the benefits it can bring when used in the marketing space and beyond.

As marketers become more comfortable with technology and the use of AI applications expands, a brand definition still requires human intervention for maximum creativity. AI is well-suited for testing and simulations, but it has not yet become a replacement for a human’s smarts and emotions.

Defining ROI
ROI is a simple way to measure an investment’s profitability, showing how much money was made compared to how much was spent, usually as a percentage (i.e. 10% or 200%). It tells you the “bang for your buck,” indicating how efficiently an investment generates earnings relative to its cost. This helps you compare different options. In short, ROI is a financial ratio comparing the gain or loss from an investment to its cost (Fig. 1). But how does one define the “costs” of AI?

AI in banking and finance is certainly earmarked for the future, but how will that be measured and affirmed? AI can’t succeed on its own, and it isn’t just a passing trend (like 3DTV). AI is intended to aid in the support of risk management, customer service and operational efficiency by continually cross-analyzing data sets across all elements of the enterprise and providing insightful information about changes, all the while running evaluation models that curate data on projects, sales, costs and other elements needed to make assertive decisions critical to success across the organization.

In 2023, McKinsey & Co. said banking was expected to be one of the top two industries spending the most on AI: “The economic potential of generative AI…is the next productivity frontier.” That report affirmed that one must “first look at where business value could accrue and the potential impacts on the workforce.”

AI has permeated our lives “incrementally,” and not just in terms of tech (i.e., from smartphones to self-driving automobiles). Generative AI apps such as ChatGPT, GitHub Copilot and Stable Diffusion have not only captured imaginations, they’re now “routine” in nearly every task from paying bills online, to ordering prescriptions, to classifying data at all segments of the population, to daily workforce tasks. Fig. 2 shows some of the AutoGPT principles and tasks that will be explored as this column continues.

GenAI “has the potential to change the anatomy of work, by fundamentally augmenting individual workers’ activities. Current generative AI, coupled with other technologies, has the potential to automate work activities that absorb as much as 70% of an employee’s time “today.” That ROI can immediately be equated to dollars saved or performance increases per unit time.

Earliest Adopters
According to a Google ad from February, 86% of marketers are using AI. A Harris Poll from October 024 states, “marketing is the most communication-driven and content-heavy function in any organization.”

As AI reshapes how work gets done, marketing teams have been among the earliest adopters, leveraging ROI through more complex communications, refined brand messaging and greater workloads. AI offers critical opportunities to reduce inefficiencies, scale content production and improve cross-team communications, often without direct human intervention.

Generative AI isn’t just another technology shift — it’s a fundamental transformation in how enterprises operate. The best CIOs are using it to drive innovation, competitive advantage and efficiency. Without AI, they risk falling behind or worse, becoming obsolete.

Researchers agree that AI’s positive benefits for marketers with 92% saying that they saw a reduced workload, 91% saw an increase in productivity and 91% reported increased creativity, with 87% reporting improved communications.

To sum it up, according to the SOBC Marketing Report, an average of 27% of marketers use AI for drafting, 25% for responding and 40% for enhancing communications—with a mix of 23% using AI to respond to emails and 28% using it to respond to chats or simple “pings.”

These are monumental changes to how the workforce is evolving, considering that you, as the recipient of these responses from AI bots, now have no idea who really responded, or if that person had some tech agent doing their work for them. Does this trend worry you or annoy you? One must now fully flesh out the integrity of the workforce and where it is heading, in whole or in general.

What is the “path to efficiency and effectiveness” now? Will it lead to more errors or reduce the risks of having little to no—or less—human intervention or thought processes involved in decision-making?

One might be reminded of “War Games,” the 1983 Matthew Broderick movie, in which computers were the players in a global epic event where AI almost left the “human decision-making process” paralyzed. Only time will tell!

Agents or Bots?
Most are familiar with chatbots, but aren’t as familiar with AI agents. But AI agents (i.e., Large Language Models, or LLMs, that perceive their environment) go well beyond chatbots. Such AI agents will plan and make decisions, run tasks and team up with other tools to aggressively attack tasks such as cross-coding, other AI services, content, operations and more.

Bots are more or less an input device that sends queries to preprogrammed sets of sequences, doing little “interpretative thinking.” Chatbots are reactive, conversational tools designed to answer questions based on scripts or AI models.

Are the emerging chatbots becoming tomorrow’s assistants, or will today’s assistants seem to be stuck in the waiting room of productivity? It’s hard to tell, but certainly worrying to some.

The structure of an AI integration into business is something we’re starting to hear more about as the term “AI” creeps into every corner of the workforce and workplace. One of the elements (aka “platforms”) that is a major controlling factor, as in any software implementation, is known as “orchestration”—and aids in managing the AI solutions, no matter the size or scale of the system.

Fundamentally, an AI orchestration platform goes beyond just simple software integration.

Every major software and cloud company is adding AI agents (generative AI, or “GenAI”) to their platforms—claiming, marketing-wise, that their solution will transform productivity and accelerate growth. But not controlling or managing coordination (as in “orchestration”) may result in having multiple disconnected agents from different vendors that can lead to confusion, security risks and inefficiency, delivering little real value or scalability.

Standards
An orchestration, when framed to include AI, is the connected end-to-end (“E2E”) application of GenAI tools. “AI agents” and automation should rationally extend across workflows, teams and systems. As more AI integration is implemented across all sectors of industry, it is important to apply some level of enterprise standards for consistency and uniformity. The problem is that there are precious few standards in place for AI, whether for integration, validation, authenticity or orchestration.

For example, platforms that use tools such as Model Context Protocol (MCP), external AI tools and agents (e.g., Microsoft Copilot, Salesforce Agentforce or even standalone ChatGPT/Claude instances) can enable and leverage a vast library of searches, actions and authentications. That means relevant capabilities can be used outside of a single MCP, integrating other AI systems. This is a significant architectural shift from Integration Platform-as-a-Service (iPaaS) tools, which are typically about connecting systems exclusively within their own framework.

What Is an AI Agent?
An AI agent (see Fig. 1 for examples) is an autonomous system that perceives its environment through sensors and then reasons, makes decisions and takes actions to achieve specific goals using actuators (functions with “calls to action”). AI agents can operate with a high degree of independence, breaking down complex tasks into smaller subtasks and adapting their strategies over time through large language model learning (LLM).

This workflow structuring is one of the core functions of artificial intelligence as we know it at this point in time. The ability to take a large or small set of complex tasks and break them down into smaller “chunks,” then reassemble them to achieve answers (i.e., “conclusions” or “results”) and produce a single-thread solution applicable to the needs of the inquiry is one of the more prominent capabilities and practices of AI.

What Is Overlooked
Of key importance to the constructs of AI, often missed or overlooked by the media or AI naysayers, is this principle of slicing any task into small enough pieces that it can be “worked on” by compute platforms that leverage a large knowledge base of data that relates to solving the issues (equations, ideas, concepts and such) that pertain to the specifics of the inquiry.

The growth and dimensions of where and how AI will meet future needs is beyond comprehension, which infers that we don’t really know just how successful a system will be and how, if or when it will make sustainable impacts on the workforce. But we do know that without the principles of orchestration, the success of GenAI and associated analytics becomes constrained and less satisfying.

According to Data­bricks, a leading data-intelligence platform, 99% of global enterprises will be using GenAI by 2027. Many may struggle to scale their projects or find gaps in infrastructure and data integration that can lead to inaccurate or irrelevant results, potentially limiting AI’s impact.

Selecting the right AI agents will ensure your choices in GenAI applications are accurate, scalable and tailored to your business needs. Structurally, the enterprise must still learn and appreciate that standalone models aren’t enough for AI success. The overall leverage part is the need to properly orchestrate systems that will ultimately allow AI agents to free up time and resources for important and new strategic work principles and concepts.

Without outside integration resources, the enterprise will find that AI platforms deliver less accuracy, are more domain-specific and, depending upon the application, are less functional when employing multiple autonomous GenAI outputs. (See Fig. 2 for a graphic representation of AI agent functionality.)

Getting More Work Done
AI agents can integrate with your existing data and applications, to get more work done while by reducing the number of errors in a project; improving efficiency (yielding less time to market); automating repetitive tasks undertaken by employees, such as customer-
service representatives, project managers and accountants; and reducing the time it takes to onboard new customers—while simultaneously generating better and more usable, trusted data.

Agentic AI apps (aka “GenAI”) can seamlessly integrate user experience, autonomous process execution and AI-powered data products to drive real business outcomes.

Business and team categories where data analysis would typically leverage GenAI include business and industry (“BI”), finance, IT, marketing, sales and operations (“ops”). Prescribed specific templates, driven continuously by your data, can be displayed across multiple interfaces or devices in graphical or tabular/columnar forms that simultaneously conform to business practices.

However, accessing, integrating and leveraging that data isn’t always straightforward. Results can help to drive better outcomes (“decisions”) by unlocking the hidden but real power of the data you collect automatically.

For example, in news media segments, producers and newsroom managers likely need to know what local competitors are showing live on the air, as well as how successful their own stories are in terms of viewer demographics, the length of time they stayed connected to a story and any ancillary information (e.g., comments or if additional searches of specifically related topics were conducted).

By continually scrubbing competitors’ live TV screens, capturing the textual lower-third info and tracking when and where a story runs in real time, teams can build a more complete picture of performance. And when a story has a “link” or QR code “to see more information,” data can be collected and corroborated as to when the link was utilized, by whom, how much information was accessed and within what time period the viewer engaged—as well as verifying the authenticity of advertising links for sales purposes.

This is all complicated information gathered from a huge volume of data sources that cannot be collected, calculated or categorized by humans in real time, as was attempted less than a decade ago. Today, this information is timely and very important when making additional content decisions for an upcoming show or a web page with follow-up information. Today’s Nielsen ratings or overnights are seldom useful in a live or breaking-news situation.

The Power of AI Augmentation
The previous examples are not about where AI is replacing a job or task—these are things even a dozen or more staff members could not reliably or continually do, especially at all hours of the broadcast day or overnight. Nonetheless, humans will ultimately make the appropriate decisions, with or without bias or compromise—and those tasks are likely to remain this way for years (maybe decades) to come.

Many up-and-coming software solutions providers are unleashing data “transparency” tools that can be customized for the needs of business and industry. This ensures specialists inside the enterprise can continually improve a company’s performance across various business segments. This is why AI, in general, is becoming so important to senior leadership and division workforces regardless of the industry they support. Faster results, more efficient practices, more trusted communications and better performance—this is what AI is coming to be and is really about.

Cloud service providers are expanding and broadening their business—and especially their service offerings—as everything seems to be moving toward AI. To help accomplish this, without rebuilding or replacing existing infrastructures, the new term “AI cloud” becomes a service offering that leverages some of the new and advanced telecom technologies the cable industry has been utilizing for a few years now.

Wondering how new service providers, including the cable industry, are effectively linking AI into cloud computing? This article will explain some of those new advances and how they are being melded into the cloud.

Don’t be surprised if, in the next five years or sooner, we start to see AI and/or cloud computing services being added to our cable systems for consumers, home and business. You’ve likely seen TV ads that exemplify the foundation of such services and may not even realize it is “already” happening.

For the future, one approach that will be needed and become of value to users is the integration of AI tool sets and their models into cloud infrastructures for tasks such as (home or business) automation, model training and data analysis. Such prospects will involve cloud platforms that leverage AI-specific resources, and in turn, can help streamline the AI development life cycle and deployment. Cloud becomes a very significant part of this AI movement.

Several new steps and approaches will provide unique opportunities for the more traditional segments of the media and technology industries and include some of the following issues, changes and conclusions.

Access to Expertise
Cloud service providers are beginning to change their physical data and transmission centers. They are investing heavily in AI topologies for end users and those with business-related requirements in such areas as operations, research and development. As the major cable providers continue to add new service segments (e.g., business connectivity and cellular communications), there becomes a great new need for these media providers to edge towards becoming cloud vendors that will, in turn, provide user access to emerging AI tools and technologies.

These providers will need to expand and provide sufficient compute power and resources, leading to the building of newer and larger data centers with the specifics and needs for AI applications. This further need for more power to support cooling and servers will force these providers to new locations, such as more rural areas where power is affordable and space is not at a premium. Connectivity to end users (customers) will further require additional high-speed (fiber optic) services which, in turn, means a change in physical distribution platforms.

Virtualization
Regardless of today’s core functions, data centers will soon need to be capable of providing new insights employing systems such as virtual cable modem termination systems (vCMTS) and Distributed Access Architecture (DAA). vCMTS is a software-based CMTS, while DAA decentralizes the cable network by moving functions from the headend to the edge of the network, closer to the customer. The vCMTS replaces the traditional, hardware-based CMTS with a software platform running on servers and includes more centralized management and control of the network (Fig. 1).

Some of the early advances made by cable operators and vendors in virtualizing the industry’s access networks and expanding their capacity could become a hybridized means for multipurpose cloud providers who are increasingly moving to virtualize hybrid fiber-coax (HFC) access networks as these next-gen technologies emerge.

Such a change is critical to meeting the challenges of emerging GenAI solutions to deploy and manage the mass data-distribution industry’s long-term health and competitive prospects. Cable service providers are always using new systems technology, such as AI, to mitigate failures. For example, one methodology is limiting no more than just a single fiber deep node issue per 40 customers. A legacy cable hub population of 20,000 to 30,000 customers using a traditional CMTS was the expectation less than a decade ago.

Moving to the Edge
In another way, the models being used by the cable industry are taking key access network functions out of the traditional cable headend and placing them in software running “at the edge” on much simpler and less costly commercial off-the-shelf (COTS) servers. Despite a slow start, cable operators are increasingly moving to virtualize their hybrid fiber-coax (HFC) access networks as the next-gen AI technology emerges as critical to the industry’s long-term health and competitive prospects (Fig. 2).

Benefits for operators include addressing the future needs to meet the capacity and connectivity challenges of a new decade. Whether the systems are to be used for traditional media distribution or as an augmentation to offerings that include “smart features” driven under the “AI umbrella,” speed, efficiency and continual access will be essential to the overall success of the mainstream media and cable solutions providers.

Cloud Platform Changes
Additional related practical requirements and capabilities for cloud platforms headed toward AI services include these core requirements:

Scalability: To rapidly and efficiently—based on fluctuating workloads—expand services without overspending on resources.

Speed and Agility: Cloud-managed abilities to provide rapid deployment and development capacities, allowing for accelerated time-to-market especially for AI applications.

Data Management Enhancement: Cloud platforms must offer robust and stabilized data storage and compute management capabilities. For AI applications, this means the cloud provider must be able to securely store, process and deliver data to customers in large volumes, especially when offering AI model training and insight.

AI Models With Prebuilt Solution Sets: Cloud AI platforms must further be able to offer “pretraining” AI models for concepts such as image recognition, LLM (large language models) and speech-to-text/text-to-speech translation.

Outsourcing Automation/Security
Process automation and optimization is a normal and routine process for AI solution systems; thus, the AI cloud platform providers are tailoring their systems to help streamline tasks that have semi-consistent applications (such as delivery of services or products and order entry prediction requirements). Optimization of these tasks, in turn, will improve accuracy and reduce costs across myriad common business functions.

AI-cloud platforms may also help or be used to improve personalized customer interactions, enable better, more efficient customer service and streamline workflows. We’ve already recognized that the “bot” is the new customer service representative!

Last but not least, AI-cloud platforms should offer robust security features.
Real-time threat detection and (automatic) compliance monitoring will aid businesses in protecting not only their sensitive information, but also in ensuring customers are clear of potential bad actors—both of which may be essential to meet regulatory requirements.

Public Interaction
AI-cloud platforms are becoming increasingly important, and we now see them expanding focus on their core competencies. Businesses—including media-centric enterprises—that must interact publicly can now outsource AI infrastructures and management to their cloud providers as part of a regular service offering. This dramatically changes the core capabilities and pushes new strategic initiatives without a need to retool and invest in additional secondary services outside their core directives.

July 19 marked one year since what has been described as “the largest IT outage in history” when cybersecurity company CrowdStrike updated its software that led to roughly 8.5 million systems crashing and refusing to properly restart.

A number of broadcasters were significantly impacted by the outage, including the U.K.’s Sky News and Sky Sports News. Sky News went off-air for parts of the morning on July 19, 2024, before returning with a much-changed backdrop, no graphics, autocue or packages.

In the United States, station group E.W. Scripps Co. told TV Tech that a strategy put in place over the past 10 years was able to quickly muster technical resources and identify and correct issues when they arose.

To mark one year since the outage, TV Tech and sister brand TVBEurope asked key media figures if they believe the industry has taken sufficient steps since then to prevent similar disruptions in the future.

Tim Claman, chief technology officer, Avid
One of the big learnings from that outage was that security strategies need to be more proactive and adaptive, rather than just responding to incidents after the fact. Since the outage, we have seen wider adoption of strategies that combine prevention and resilience methods.

Customers are bolstering traditional detection methods (like antivirus, EDR) with technologies that more dynamically identify potential threats, including AI-based monitoring, zero-trust models, dynamic policy management and more rigorous update testing, to help prevent and minimize impact from security incidents like we saw last year. While a lot of the technologies and approaches (CNAPP, IAM, EDR, Endpoint Isolation, Zero Trust, etc.) are not new, they are now being used more intelligently in combination with operational best practices to reduce risk.

Beyond the technological improvements and the evolution of best practices, we have seen a shift in mindset. The outage was a wake-up call for our customers, as well as for the vendor community. Media enterprises realize that they need to own their own destiny on security, rather than trusting security vendors, so they are more proactively managing their security strategies and practices. As a technology vendor, Avid is receiving more detailed and thorough security questions and requirements in RFPs, a reflection of our customers’ more intensive approach to security strategy.

Rowan de Pomerai, CEO, DPP
I think it’s probably fair to say that no, not enough focus has been given. There’s been a huge amount of business transformation and disruption, and AI has taken so much of the technical limelight, so it’s been easy to lose focus on concerns like security.

But perhaps AI could be the solution as much as the problem: contributors to the “DPP 2025 Predictions” said that “security concerns will go beyond human scale” as the capability to keep up with ever-changing threat vectors will become so complex that AI and other automation tools become essential.

Our “State of Media Technology Security” report also exposed a gulf between customers and vendors when it came to their assessment of the security of modern media technology tools. Clearly, there is more collaboration needed as we move forward.

Neil Maycock, TVBEurope contributor and business adviser
I think the CrowdStrike incident really highlighted that trying to protect against the ever evolving threats in cyber security carries its own risks, and the fact that major operations were so severely impacted is a testament to that. These were companies who take security very seriously and will have had extensive protocols in place to keep systems safely up to date, and yet an unforeseen scenario had devastating consequences. Therefore, to answer the question have sufficient steps been taken to prevent a repeat, I think the answer is of course industry will have protected itself from another CrowdStrike, but the real question is it ever possible to anticipate all possible scenarios?

Cybersecurity is a classic risk management exercise, where risk must be balanced against cost, both financial and operational impact. If we lock our systems down to the extent it compromises the ability of our organisation to operate, then the cost may outweigh the potential risks. Quantifying the risk and cost impact is a massive challenge, and one that is getting harder all the time. We frequently talk about the impact of AI in the media industry, but across all sectors AI is now being leveraged to implement ever more sophisticated cyber-attacks. For example, AI’s ability to impersonate key personnel, or create very specific personalised phishing attacks is creating new challenges.

Ensuring companies are adequately protected is an unenviable challenge.

Karl Paulsen, TV Tech contributor and retired CTO
I suspect this won’t be the last time we see something like this happen when there are so many moving parts in a single system. With Windows 10 support evaporating, much of the base software for users will be shifted and everybody and their dog will be wondering how to address a significant change in their hardware and software (including third-party support).

Impacts on multi-cloud will need to be managed, and it is still unclear what “AI” is going to do to everyone’s architectures. Errors are still likely to occur and an overload of systems management is a potential liability.

Dan Pisarski, chief technology officer, LiveU
The notorious CrowdStrike outage in 2024 taught an important lesson about shared vulnerabilities: you could have an entire, physically diverse set of servers and workstations—even in a building designed to withstand disasters (what if a meteor falls on the building, after all?)—but if every one of those servers runs CrowdStrike, you’re still at risk. This event has put real pressure on media organisations to rethink what disaster recovery looks like, especially when it comes to intentionally building diversity into Disaster Recovery (DR) plans.

Is your primary production on-prem? Then your DR should be in the cloud. Is your playout path based on a local fibre run? Then your DR plan should use wireless. The goal is simple: if something happens that wipes out an entire class of options (as the CrowdStrike incident did to Windows PCs and servers), you need a diverse DR plan that can avoid the fate that takes down not just one component, but an entire class of your mission-critical systems.

Cloud-based production platforms provide a diverse form of backup to hardware-based production solutions running Windows. If there is another day when “all Windows PCs don’t work in the world”, then you have a backup plan. The cloud solution does not require integration with Windows, and while it is not an exact match 1:1 of features of a hardware-based solution, that matters less when it is becomes the primary solution in a disaster recovery scenario.

This kind of built-in diversity applies at both small and large scales. At the small scale, for example, your bonded-cellular active/active wireless transmissions should use multiple carriers—not just for performance, but also “just in case” one carrier experiences a localized or nationwide outage (it’s happened!). On a much larger scale, diversity might mean having an elastically scalable cloud production service ready to go, even if you “usually” rely on on-prem production.

Many organizations use the cloud to augment traditional data center capabilities. Such complementary uses often include storage, data processing and records management, such as HR or taxes.

According to World Wide Technology, 90% of business organizations will leverage the cloud at some level. Half or more of businesses surveyed said they are shifting IT budgets to cloud, and nearly two-thirds are already at some level of cloud deployment.

Cloud Stall
All these efforts mean workflows are rapidly changing, despite the fact that cloud migration can be complicated. When shifting to the cloud or an adoption initiative slows down (or, in some cases, grinds to a halt), this is referred to as “cloud stall,” which can be debilitating and embarrassing across the board. Therefore, adopting a “cloud strategy” is essential to mitigating the potential impact of cloud stall.

Missing the planning portion of cloud implementation often results in misconfigurations and usually leads to suboptimal cloud deployments. Eventually, the organization will find itself needing to backtrack and correct various errors—sometimes with lengthy downtimes or costly reworks.

Cloud Maturity Model
The modern digital transformation, for business or private purposes, has almost completely shifted to the cloud. Successfully reaching this transformation—whether you’re new to cloud or you’ve already invested in a public, private or hybrid deployment model—could very well depend on your ability to systematically reach and mature your cloud capabilities.

A “cloud maturity model” (CMM) is a framework that helps organizations assess their current cloud capabilities and identify areas for improvement as they adopt and utilize cloud services. This model (Fig. 1) outlines distinct stages of progress, each with increasing levels of cloud usage and optimization; you start out small and add capabilities until the model develops into a mature, functional solution that meets the needs of the business and its activities.

A cloud solution is not an overnight activity; it can and often will require many iterative processes or stages. Technologies available today may become inefficient or even obsolete during workflow development and utilization.

Given the broad reach and capabilities of the varying technology stacks, it can be challenging to properly and efficiently harness the cloud’s potential. Many who “go at it on their own” find themselves making mistakes such as:

1. Lack of Adequate Cloud Security: As the organization rushes to migrate its applications to the cloud, it may find the oversight of robust security measures ends up exposing sensitive data to potential threats, including data breaches, unauthorized access or even service disruptions.

2. No Proper Strategy for Monitoring and Maintenance: Failing to establish proper protocols, including monitoring tools and maintenance procedures, can result in performance bottlenecks, outages and a compromised user experience.

3. Inadequate Backup and Disaster Recovery Plans: Envision an engineer who assumes that data stored in the cloud is inherently secure. The team, therefore, doesn’t implement robust backup procedures. Actions, including accidental deletions, cyberattacks or system failures, could expose the absence of a backup plan, which could result in irretrievable data loss.

4. Insufficient Employee Training: Lack of sufficient and proper training is a common cloud deployment mistake that cloud startup managers often overlook. Administrators and managers might assume a team can rapidly adapt to new cloud tools and services without comprehensive training. Such misconceptions often lead to a significant gap in skillsets in various segments of the business units and staff.

5. Not Understanding Cloud Costs: This is a common mistake and will eventually result in a costly impact. Teams often don’t recognize costs and may overlook the nuances of cloud pricing models, underestimating the financial implications of their deployments. This can result in a cycle of reactive cost optimization, i.e., scrambling to rein in expenses after they’ve spiraled out of control.

6. Rushing Cloud Migration: The excitement around the “cloud promise,” such as improved scalability or flexible agility, could lead teams to not thoroughly assess their readiness for the cloud environment and, in turn, hastily migrate applications with improper parameters or useless end results.

By rushing into cloud migrations, poor results with cost overruns may occur. To mitigate this possibility, engineers and startup founders should prioritize cloud implementation processes using a comprehensive planning procedure, sometimes provided by a third party skilled in the steps and stages of cloud implementation for your business type.

7. Improper Resource Allocation: A common stumbling block for engineers, administrators and startup founders is not providing adequate resources (people, materials, testing, etc.). This could also include the overprovisioning of server resources in anticipation of much higher demand than materializes in reality. These startup errors could risk losing customers and damaging your reputation as users experience a subpar application performance.

Mitigating
“On average cloud deployments don’t go well,” as was quoted in a CIO article in 2015. Back in those days, “70% of the implementations required changes,” “43% of the cloud projects failed or stalled” and “close to half required an increase in budget within six months,” according to THINKstrategies and INetU surveys on enterprise migration to the cloud.

Things have changed dramatically in the past decade and success rates are much higher now. Nonetheless, mitigating common cloud deployment mishaps is essential for ensuring the success, security and efficiency of any cloud-based implementation. Fig. 2 shows some examples for consideration that might make sense when looking to move your organization to the cloud or “re-architecting” an existing platform into a more mature solution set.

Whatever your situation, look thoroughly and plan appropriately for both the immediate and long-term activities your organization must address and seek advice accordingly to alleviate risk and failures.

Using artificial intelligence (AI) for practical purposes is gradually making it into the mainstream public world, despite the fact that it has been in regular use for many “business & industry” purposes for probably a decade. While the media seems to be adding the term “AI” as the resolute action for many things such as graphics, text creation, false or misleading anything, sadly AI is getting a bad wrap in many circles.

This may, indeed, be because AI is so misunderstood—which, frankly, is why my TV Tech colleague John Footen and I created these monthly columns on this rapidly advancing and maturing technology.

This month—in the first of two parts—I’m going to go back to some fundamentals about what it takes to use AI (specifically “generative AI”) by providing some overviews graphically and textually to kick-start “what you need to know about AI” to uncork the genie into practical use and applications. We need to go back a few years and start to interleave other sets of terms to place the bigger picture into a better perspective.

For the First Time in Forever
For the first time, we can now actually “ask data a question” in human speak/text and get an “original” answer as complete content, returned in human text as speech or as a visual interaction. However, these changes are not without a price—in power, real estate and infrastructure change. Some examples follow.

For perspective, “a single Google search can power a 100-watt lightbulb for 11 seconds,” says Bill Kleyman, CEO of Apolo, an AI platform and infrastructure company. When you shift to new AI solutions like ChatGPT, this multiplies by many fold—which means the computer is 600-800x more powerful than Google search, leading to as much as a 600-800-power draw increase.

Thus, it takes a lot of data to get AI to function, which is why we hear that the prominent players (Google, Oracle, Amazon, Microsoft, etc.) are building gigantic warehouse-like data centers—the scale of multiple sport stadiums—in areas where megawatts of power are available and land is spacious enough for extended growth well into the future.

Big Data
You’ve likely heard the term “big data” for quite some time but only wondered what it really means or how the term is applied. In the early days, not many realized the value and importance of “big data” to the concepts and principles of AI. Even today, the mention of “big data” is not applied much when speaking about AI in the general sense. But without “big data” and its associated corollaries (see the glossary sidebar)—we could not have effective, consistent, believable, and trustworthy AI.

AI needs lots of data to train its systems on, which is how large language models leverage massive data collections to build a “model” on which to train the systems. An LLM will be used to generate a “consensus” from which to output an answer, but this takes much more in infrastructure than what we’ve seen in the past.

Power Density
According to the 2024 AFCOM data center report and presentations “Key-Trends-and-Technologies-Impacting-Data-Centers-in-2024-and-Beyond,” based on data from 2021, a datacenter rackhousing “IT-centric” equipment averaged around 7kW per rack; but we have already arrived at average rack densities of 12 kW equipment.

For an AI complex, that is nowhere near enough power to contain the kinds and types of compute systems that AI demands. According to the report, “the upward trajectory to [will] continue with 20 kW averages likely by 2030.”

Landscape
The Generative AI (GenAI) application landscape is a moving target, constantly growing and shifting across all industries but specifically in areas relative to: (a) imaging, (b) video; (c) text; (d) research; (e) 3D: modeling/manufacturing and medical; (f) speech: translation; and (g) code: i.e., documentation, web app building, text to SQL (like) databased, actual self-created code-generation. Fig 1 in the sidebar outlines areas where AI is being applied in everything from research to manufacturing to communications.

AI Capabilities
GenAI can be used to alter the “tone” of a segment of content, based on the audience it is to address. In essence, by altering the phrasing and the content, the text, visual details or audio can be adjusted (slanted or enhanced) to fit the audience it is being supplied to. Such alterations may include “setting your voice” i.e., adjusting your generated text to sound. Prompts may include (a) suggest and construct counterarguments, (b) improve the emphasis on a certain section, or (c) to regenerate new text for an audience that has not previously used a new or differing experiential design.

Web design is one which can be dynamically changed based upon the “user experience” or by leveraging the data secured from previous searches that the web user has looked at or searched for. This is very common on search engines whereby the user is “pushed” into sponsored areas of a document (webpage) that would be attractive to the user and promotional for the advertiser.

Heretofore, this was not possible without GenAI technology, but now features such as “intuitive navigation” or “responsive design” or “visual hierarchy” which include new “interactive elements” (Fig. 1), can be used tp automatically generate and create “user-specific” pages that fit the both the sponsor/advertiser objectives and the expected needs of that user. This is an important part of how AI is pushed into modern media.

Visualization
Visual hierarchy is a technique whereby graphic elements are resized or adjusted to drive the user’s attention to more important elements on a webpage. By capturing user data from previous searches or by leveraging “big data” from a machine learning process, the tone of the content may be “swayed” to best fit the specific user-type or perspective.

Gen AI is used to make the user “want” to look deeper or find a media element more pleasing—including altering the audio (musical tones, speed, intensity, softness, etc.,) or speeding up or better emphasize the verbal rates of the text or by creating art which is typically desired (or found) by similar users as a particular type of said art or video content.

This became more emphasized by changes which the original owners of the 007-franchises, whose IP is now owned by Amazon. One of the expected “values” to be created by Amazon will be the use of GenAI technologies to create new stories, new content, and/or the selection of new talent who will attract more and consistent “Bond-Enthusiasts.” All of this, despite Amazon’s CEO Andy Jassy’s denials that “he does not envision that AI is going to write a Bond movie any time soon.”

Greater Depth
By continuing to improve or elaborate on the content, users may then add more questions or comments to those replies created by the prompts. However, one should be cautious: Users could “overcompensate” the GenAI bot replies with too many follow-up inquiries. The results could be what is known as “hallucinations,” whereby the bot begins to make up artificial answers which may go off-base in both topic and reaction.

Users will then learn just how far to push the GenAI bot, and where the answer has received enough iteration or depth to “call it quits.” This is one of the areas where researchers are worried that much greater time with AI will be necessary for complete confidence to be achieved.

Watch for the conclusion of this topic in the September issue of TV Tech.

One of the lesser-realized but very important elements of artificial intelligence is real-time adaptation and decision-making. Where is this important, one might ask? The ability to process information as it arrives and then to make informed decisions without significant delay is an area where AI can be quite valuable.

Familiar applications of real-time adaptation include command-and-control environments, security situations, traffic control or monitoring environments and in autonomous driving (autopilots). Every one of these situations requires intelligent systems to be able to make adjustments in response to dynamic situations and—in most cases—in real time.

Adapt in Real Time
Any one of us can probably imagine the number of decisions that must be made in a self-driving vehicle solution. The ability for any system to “adapt in real time” is becoming essential in this fast-moving world—and AI is a primary element in those advancements.

Today, companies must be able to act quickly on data-driven insights to be more agile, proactive and to seize emerging opportunities or respond to sudden market shifts. Amazon is a good example of a business that must be able to move in a certain direction without being burdened by “legacy” components that bind it to restrictive methods that cannot react to sudden changes in marketplace demands.

For time-based analysis, the AI-driven environment might depend on the following: (1) continuous time analysis and (2) discrete time analysis. Each of these specific methodologies in mathematics, networks and analysis have subcomponents that become applicable to many elements in media, business/financial forecasting, signal and process management and system modeling (both complexity and accuracy).

Signal processing involves elements such as signal data visualization techniques, preprocessing and filtering techniques, plus physical-based time-domain and frequency-domain analysis (especially in real time), and the use or application of the data derived from signal processing.

Broadly defined, signal processing is a fundamental discipline in data science that deals with the extraction, analysis and manipulation of signals and time-series data. The depth of this science can get very complex and highly dependent, which is why we may see “data scientist-engineer” as a profession grow rapidly in the workplace.

Reading the Signals
In data science, a signal is defined as a gesture, action, element or sound used to convey information or instructions. From a third-person perspective, signals “transmit” information (such as instructions) by such means—i.e., by gesture, action or element/component, including audio/visual elements such as sound, light or even temperature changes in the environment, etc.

When placed into the context of signal processing, a “signal” can be any form of information that varies over time or space. Such signals may take many familiar forms, ranging from audio waveforms and temperature readings to financial market data and sensor-activity measurements. AI operations function on categorizing such signal data forms and learning the variations or changes from actual environments prompted by stimuli generated by external components including humans, the climate, physical alterations or altercations and such.

Neural Network
According to IBM, a neural network “is a machine-learning program, or model, that makes decisions in a manner like the human brain.” In our cases, these networks are specifically a computer system modeled on the human brain and nervous system, i.e., the “ideal” AI-environment. By using processes that mimic how biological neurons work together to identify phenomena, the model can weigh options and arrive at conclusions.

Conclusions are generally reached by using a series of training exercises which in turn “machine-learn” to improve their accuracy over time. As these successive training exercises are fine-tuned for accuracy (and application), they become powerful tools in data and computer science and, in turn, support artificial intelligence. The results are that tasks, such as image or speech recognition, can take seconds compared to the hours a human might require using manual identification methods.

One of the best-known examples of a neural network is Google’s PageRank (PR) search algorithm, used to rank web pages in its search-engine results tabulations. We note that PR is named after both the term web page and Google’s co-founder Larry (Leonard) Page—associated with the co-founder of Google, Sergey Brin.

ANNs and SNNs
Neural networks are sometimes cataloged as artificial neural networks (ANNs) or simulated neural networks (SNNs). There are several types or forms of neural networks, two of which are discussed here.

Artificial neural networks (ANNs) are a type of machine-learning algo­rithm that employs artificial neurons—a network of interconnected nodes (see Fig. 1 for a conceptual node diagram). These nodes then attempt to model the human brain’s neural network. Each individual node acts like its own linear regression model—composed of weights, a bias (i.e., a “threshold”) and an output. Linear regression models predict the value of a variable based on the value of another variable (see Fig. 2 for equation).

Linear regression fits a straight-line model (or surface) that is useful to “minimize discrepancies between a predicted value and an actual output value”—the approach in the training models used in artificial intelligence solutions and appears much like the slope equation from Algebra 1 (y=mx+b). For more information on the mathematics of these principles, follow up on the details of linear equations, least squares methods, and predictive coefficients.

Typically, ANNs will be used to solve complex problems—for example, facial recognition or document summary processing. Essentially, the theory behind the ANN is the teaching of computers to process data in methodologies that mimic the human brain.

Disadvantages of ANN may include: (1) they are computationally expensive and consume massive amounts of training cycles to obtain accuracy; and (2) it can be difficult for them to perfect predictions or categorize data. At this point in time, generative AI (one of the more familiar AI activities) can be considered “experimental” at best—with improvements being made as persistence in applications and libraries of training models are created.

A simulated neural network (SNN) is simply another name for an artificial neural network (ANN), considered a subset of machine learning. Summarily, ANNs are made up of connected nodes, or artificial neurons, that are loosely based on the neurons in the brain.

Discrete and Continuous
In our AI category, “continuous time analysis” refers to studying systems where changes occur smoothly over an uninterrupted time interval. Contrary to “continuous” is “discrete time analysis,” which examines systems where changes are only observed at specific, discrete points in time, essentially treating time as a series of intervals rather than a continuous flow.

The nature of the problem to be solved and the degrees (amount, type and frequency) of data being analyzed are determining factors when using either the discrete- or continuous- time analysis approaches.

Discrete time models often are the more preferred choice due to computational ease, however, continuous time models can provide a more accurate representation of certain real-world phenomena when applicable (e.g., in self-driving vehicles or other autonomous activities.)

AI Art and Approaches
AI allows leaders of organizations to make better decisions by using the built-in methodologies employed in many conditioned AI approaches to problems (such as linear regression techniques.)

Furthermore, better insights into the solution may be accomplished by uncovering patterns and relationships that others might have previously seen and thought they already understood. Fundamentally this is how AI is utilized in the generation or modification of art and images—referred to as “AI art.”

AI art is “any kind of image, text, video, audio or other kind of digital artwork produced by generative AI tools.” Such tools leverage millions of written, visual or aural content samples in reference to the prompts or known images employed when creating AI-generated art. AI art is currently integrated into many, if not all, of the major products from companies including Adobe, Microsoft, Google and more.

Throughout the 20th century, knowledge has continually expanded, stemming from the evolution of eras such as the industrial revolution, the space program, the atomic-bomb and nuclear energy and, of course, computers. In some cases, it may appear to the masses that artificial intelligence is about as common as a latte or peanut-butter-and-jelly sandwich. Yet the initial developments of AI date at least as far back as the 1950s steadily gaining ground and acceptance through the 1970s.

It wasn’t until the late 1970s and early 1980s that computer science began to emerge from a data-driven industry using large “main-frame” computational systems into platforms for everyday uses at a personal level. While the Mac and early PCs (beginning in the 1980s) were game changers, they were certainly limited on compute power and not designed to “learn” or render complex tasks with modeling or predictive capabilities.

Computers of that time relied on programming based essentially on an “if/then” language structure with simplified core languages aimed at solving repetitive problems driven by human interactions and coordination.

Trial and Error
As sufficient human resources and computer solutions began to develop and create “expert systems” (circa 1990s–2000s) the computer world rapidly moved into a new era—one built around “knowledge” and driven by language models with basic abilities to train itself using repetitive and predictive models not unlike those that infants or toddlers use to hear, absorb, repeat, say and “tune” their mind and physical stature to communicate, to learn physical principles (such as walking), coordination and more.

If you’ve ever observed a two-year old learn how to maneuver around a slick pool edge surface, make their way to the steps into a pool and gradually ease themselves into the water in a fashion that uses mental observation (aka “memory senses”), trial and error, and repetitive steps (slow walking or crawling) while a parent also observes the child’s actions to mitigate accidents or errors—you’ll quickly see, figuratively, what occurs in a set of computers or servers built to use software to train itself over time to create in part what will become a large language model (LLM) by applying its programming repetitively that eventually finds a way (i.e., to model) the original programming into a formidable model that reaches the desired goal.

Without Explicit Programming
Machine learning is just that kind of process and is the basis of AI, whereby computers can learn without being explicitly programmed. This generalization of ML has classifications that are utilized to differing degrees as diagrammed in the figure on Machine Learning Tasks (Fig. 1). Fundamentally, machine learning involves feeding data into coding algorithms that can then be trained (through repetition with large sums of data) to identify patterns and make predictions, which then form new data sets from which to better predict the appropriate outcome or solution.

A variety of applications such as image and speech recognition, natural language processing and recommendation platforms make up a new library of systems. The value relationship gained in this training process can be developed by (T) defining a task; (P) establishing a performance figure or criteria (i.e., how well did it do or how far off were the results); and (E) finding a resulting data set, the experience, given the data set it was provided for such an analysis.

Machine learning is a continual process whereby trials create results that get closer and closer to the “right solution” through reinforcement. Computers/servers learn which data set is then classified as “more right” or “less right” (i.e., more or less correct); and then stores those results that then modify the learning algorithm(s) until the practice gets as close to “fully right” as possible without “overshooting” the answer and risking a data overload or a false (hallucinogenic) outcome.

One downfall in ML is that the system may go “too far” (i.e., it has too many iterations), which then generates an exaggerated or wrong output and produces a “false-positive” that gets further from the proper or needed solution. Then one questions, “just how far does the generative process go before it is stopped?” When the system gets to the “regression” point—a case where the outputs are continuous rather than individual or discrete—then a categorizing sensing algorithm would necessitate supervisory termination or a resteering of the operation to a more dimensioned (contained) level, which restricts the continued processing.

So, in addition to the learning algorithm, there are sets of management algorithms that must be applied throughout the learning process to mitigate these so called “hallucination” possibilities. Remember the toddler in the pool, this manager may be the parent in this case, the individual who stops the child from being hurt or risking a task (T) that could be catastrophic in nature.

ML can (and is) used in many everyday solutions including email filtering, telephone SPAM filtering, anomaly detection in financial institutions, social media facial recognition, customer data analysis (purchase history, demographics) and trends, price adjustments (such as with non-incognito searching), and emerging advanced solutions including self-driving cars and medical diagnostics. The “balancing” apparatus must weigh multiple solutions, alternatives and decision points, which in turn keep a runaway situation from occurring, resulting in an unnatural or impossible situation or solution.

Industry Challenges-Bias & Fairness
Besides the rapidly developing capabilities, there are as many challenges in this evolving AI industry as there are opportunities. Data Bias and Fairness (e.g., in social media) is highly dependent on the data it has available for training. Bias can obviously lean toward and potentially lend to discriminatory solutions.

Privacy protection as well as security breaches head the users into areas that result in illegal or illegitimate practices. Given the ease in spinning up huge data server systems in the cloud, the possibility of running tens of thousands of iterations on passwords or account numbers means the risk to the customer (as in credit card fraud) can grow exponentially to the number of credit card holders. Banks and credit services use very complex AI models to protect their customers.

This in turn opens the door to another level of AI—that is risk, fraud protection analysis and monitoring. It’s a huge cost to the credit card companies, but one that must be spent in order to protect their integrity.

Another concern is in automation and the potential for job displacement. It is inevitable that some people will be displaced by automated AI solutions. In turn, these new dimensions are opening up new job opportunities across new sectors of the workforce, such as data analysis, machine learning management, data visualization, cloud tool development, laboratory assistants or managers who test and hone these algorithms.

There continue to be many misconceptions related to these new words and their actions. Buzzwords such as "machine learning," "deep learning" and "artificial intelligence" have many people thinking these are all the same thing whenever they hear the phrase “AI.” Regulations are being developed internationally and within our own legislatures that directly relate the “AI word” to machine learning or vice versa.

Most of these early “rules” are done in a pacifistic way, likely because the legislative authors have little actual knowledge or background in this area. Obviously, we risk going in the negative direction by reacting improperly or too rapidly—but things will surely happen that will need to be corrected further downstream. Doing nothing can be as risky as doing too much.

Easily Defined and Managed
As for the media and entertainment industry, efforts are well underway to put dimension on the topics of AI, ML and such. As with any of the previous standards developed, user inputs and user requirements become the foundation for the path towards a standardization process. We start with definitions that are crafted to applications, then refine the definitions that reinforce repeatable and useful applications. Through generous feedback and group participation, committee efforts put brackets around the fragments of the structures to the point that the systems can be managed easily, effectively and consistently.

For example, industry might use AI or ML to assemble, test or refine structures, which will in turn allow others to learn the proper and appropriate uses of this relatively new technology. Guidance is important, as such, SMPTE, through the development of Engineering Report (ER) 1010:2023 “Artificial Intelligence and Media,” is moving forward on this development and has a usable structure from which newbies and experienced personnel can gain value from.

Even the news media has interest in its future in AI. In their book “Artificial Intelligence in News Media: Current Perceptions and Future Outlook,” Mathias-Felipe de-Lima-Santos and Wilson Ceron state that “news media has been greatly disrupted by the potential of technologically driven approaches in the creation, production and distribution of news products and services.” l

Many organizations still operate without a defined cloud strategy or even a cloud skills plan to follow. This, among others, poses challenges to the users who wish to take the best advantages of the newest emerging trends of cloud and multi-cloud services.

According to Pluralsight, as much as 70% of organizations struggle to drive “customer value” in the cloud. Of those 70%, more than 50% say they have half or more of their infrastructure in the cloud. And ironically, another 49% say they are “actively moving more of their data” into the cloud.

The question becomes, without the needed resources that are essential to keeping the wheels on the bus going round and round, how long will it be before a potential head on collision with cloud and business be expected? And what can, should or must we be doing to cut the potential risk down?

This isn’t any longer about “migration” (to the cloud), it’s about the “rapid acceleration of and the adoption of many new services” that depend on and leverage the cloud. But a surprising percentage of these leaders don’t have a strategy, plan or approach on how to achieve these targets.

Moving From Tactical to Strategic
Cloud maturity is a hypothesis that takes the steps of implementing the “mechanics” of cloud and moves it to a more focused, lean and effective methodology of making the cloud work without overtaxing the team or the budget. Using an “ad hoc” tactical approach may resolve short-term issues, but it is by no means a methodology for the long term.

In some circumstances, leaders have become so focused on the technology of the tactical approach that they forgot to see the forest for the trees. Prioritization was lost, structure was fragile, and the outcomes were meniscal if any.

Culturalism
Your organization is likely very good at what it does, but sees and hears about the cloud, about AI, about machine learning (ML) and yet can’t see how or why it might be advantageous for the team—likely because no one on the team knows about those terms or how they might fit into their own structure or day-to-day life.

Well, my friends, get ready for a change because those previous three terms (AI, ML and cloud) are well-integrated into this new world and are making huge inroads into the next generation of tactics, strategies and operations right now (not next year or the next quarter, but right now!).

One of the first steps in approaching cloud maturity is to change/shift the culture in the organization. Many (around two-thirds) of these organizations put an emphasis on technical training of their teams, ignoring the elements associated around “cloud native” learning company-wide.

Cloud is a culture with its own language, its own “speak,” its own management, and its own sales approach. Perhaps this is why the whole concept of artificial intelligence is so challenging. The public is asking our government to place limits on things that even they (the rule makers) don’t live, breath or speak about daily. This has got to change, starting on the ground floor.

Cloud Literacy
According to findings used in the research of this article, only about 15% of most organizations’ team members understand or know much about cloud terminology. However, more than 40+% of the IT team members of those organizations surveyed have a sufficient technical understanding of cloud to aid in supporting a transition to the “new world.” A surprising amount of the IT staff and leadership staff possess a high degree of cloud understanding, with almost 75% of the IT staff being either cloud-certified or are learning cloud.

This puts the burden clearly on the educator’s communities and in getting everyone up to speed, just like learning reading, writing and arithmetic. This is priority one!

Policies, Tactics and Issues
Last year (2023) security was described as the one area that continually needs to be improved upon. It was listed as “the number one challenge” and the area with “the largest skills gap.” Most searches on cloud needs will start with security being the top issue. The other positions cloud users and administrators should look to include (see Fig. 1):

• Compliance: ISO 27001 is the international gold standard for information security management. It proves the strength of your security posture to prospects and customers in global markets. Achieving ISO 27001 certification can be time-consuming and expensive.
  
• Performance:
  Cloud performance tuning is the process of optimizing the speed, efficiency and reliability of cloud-based applications and services. It requires a combination of technical skills, analytical tools and best practices to identify and resolve bottlenecks, errors and resource wastage.
  
• Reliability and Availability:
  Availability in cloud computing is a crucial aspect of reliability, focusing on the accessibility of services to users. It indicates the percentage of time that a cloud service is operational and reachable.
  
• Data Breaches:
  Are numerous, avoidable and can be mitigated by following suggested practices in this and other documents related to security and best practices. The largest (top) in the amount of exposed data was the Indian Council of Medical Research (ICMR) in October 2023, where a threat actor using the alias “pwn0001” posted a thread on Breach Forums brokering access to identification and passport details (including names, addresses and phone numbers) of 81.5 million citizens of India.
  
• Lack of Expertise:
  Covered in part in this article, the cloud and IT skills shortage is apparently damaging organizational landscapes and is expected to impact businesses in critical areas such as not meeting financial targets, increased exposure to security risks and digital transformation delays.
  
• Security:
  Four top areas of concern include (1) Unmanaged Attack Surface; (2) Human Error; (3) Misconfiguration; (4) and Data Breach.
  
• Multiple Cloud Management:
  A set of tools and procedures that allows a business to monitor and secure applications and workloads across multiple public clouds. Ideally, a multi-cloud management solution allows IT teams to manage multiple clouds from a single interface and supports multiple cloud platforms (such as AWS and Azure) as well as new tools like Kubernetes.
  
• Portability:
  Portability and interoperability relate to the ability to build systems from re-usable components that will work together “out of the box.” A particular concern for cloud computing is cloud onboarding—the deployment or migration of systems to a cloud service or set of cloud services.
  
• Control and Governance:
  Is a set of rules and policies adopted by companies that run services in the cloud. The goal of cloud governance is to enhance data security, manage risk and enable the smooth operation of cloud systems.

As one can tell, cloud management tools for success are not unlike those found in many IT administration practices. The topics are broad, and the tool sets available from service providers are vast. The best advice this can provide will be to ensure the organization has a structure and an aligned forum inside the leadership to start today on a foundation and a plan to be ready for the “now” in cloud. Seek a consultant or a professional who understands the prospects, needs and has the capabilities to drive forward the cloud maturity needs for the entire organization. l

Karl Paulsen is the retired CTO from One Diversified LLC. He has 50 some years of experience in broadcast media technologies and is a regular contributor to TV Tech in the fields of cloud, media, IP and AI. Look for an upcoming series on AI over the coming months. He can be reached at karl@ivideoserver.tv.

In our current information/data age, the storage and protection of data is an extremely important part of any business venture. The degree of digital data being produced, globally, has long been outpacing the amount of storage available—irrespective of the medium onto which that data is stored.  

Cloud, obviously, is currently taking first place on the storage agenda given its flexibility and user expectations, however, there is a relatively new means for data storage that arrived several years ago and is now taking on some very different new perspectives.

We’re speaking of “DNA data storage”—yes “DNA” or more formally known as deoxyribonucleic acid—that tongue-twisting, nearly impossible to write term we learned back in high-school physics classes. That is, this storage concept is the enabling of molecular-level data storage into DNA molecules that leverages biotechnology advances in synthesizing, manipulating and sequencing DNA to develop archival storage. 

The next few paragraphs will take a very basic overview of the chemical make-up of elements in molecular science. You don’t necessarily need to be a biochemist to fully understand this section, but you can quickly see how this leads into the LOCO coding structure and error detection properties which are essential for DNA data storage.

The term “LOCO,” or in this case D-LOCO (for DNA-LOCO) means Lexicographically-Ordered Constrained Codes, which are “line codes that make it possible to mitigate interference, prevent short pulses, and will generate streams of bipolar signals with direct-current (DC) powered content” through the “employment of balancing.” These principles are found in magnetic-recording (MR) devices, in Flash devices, in optical recording, and in certain computer standards.

Ten Terabytes on a PinHead? 
Exploring the intersection of information technology and molecular biology has created a scientific means for the storage of over 10 terabytes of data in the space of a faint smear less than the space of 0.25 x 0.25 inches (0.0625 sq-inches). Comparatively, from American theoretical physicist Richard P. Feynman’s dissertation (December 1959), he theorized how to manipulate, manufacture, and control things on a micro-level (small) scale. 

From a coding perspective, this micro/nano technology practice dates back to work in 1948 whereby the density of data is increased through the use of such “constrained codes” which resulted in increased storage density in MR (i.e., magnetic recording). Such practices are still widely in use today which mitigate interference in today’s two-dimensional MR systems.  

Functional Artificial Objects
The science of this DNA storage remains somewhat experimental and is very much an ongoing research project of well recognized coding and by biochemistry experts throughout the world.

For example, using in-silico (i.e., experimentation performed by computer) and wet lab experiments, the Molecular Information Systems Lab (MISL) at the University of Washington (UW) in partnership with UW Computer Science, Electrical Engineering, and Microsoft Research, have brought together faculty, students and research scientists with expertise in computer architecture, programming languages, synthetic biology, and biochemistry to enable the use of DNA as a high-density, durable and easy-to-manipulate storage medium.  

Historically, the idea for DNA digital data storage began around 1959 when Feynman (see sidebar) outlined—at the annual meeting of the American Physical Society at the California Institute of Technology—the prospects of artificial objects of the microcosm and biological microcosms having similar or even more extensive capabilities in his paper “There’s Plenty of Room at the Bottom.” 

Another book worthy of further understanding is by Ed Regis (April 1996) called “Nano: The Emerging Science of Nanotechnology.” Nano tells the gripping story of how K. Eric Drexler and other scientists pioneered this emerging science. It explores what molecular nanotechnology could mean for our future as it was presented to scientists and congressional representatives on June 26, 1952.  These concepts were further recognized by future Vice President Al Gore, who presided over the presentations to the panel in 1996. 

Biochemical DNA & RNA
Functionality wise, DNA digital data storage is a process that encodes and decodes binary data to and from synthesized strands of DNA. Arguably, there is enormous potential resulting from its high storage density, but the practical use of DNA data storage is (currently) “severely limited” due to its high cost and very slow read and write times.   

In biochemistry, the depth of this topic is ginormous and well beyond the details presented in this article. However, the basics of what composes the DNA-elements include some of the following micro-biological concepts and principles:

A nucleoside is comprised of a nucleobase (which function as the fundamental units of the genetic code and five-carbon sugar (ribose or 2-deoxyribose). Nucleobases are nitrogen-containing biological compounds that form nucleosides, which, in turn, are components of nucleotides, with all of these monomers formulate (i.e., the constitute) the basic building blocks of nucleic acids.

A nucleotide is “a compound consisting of a nucleoside linked to a phosphate group.” The nucleotide is the molecular building block of nucleic acids, RNA and DNA, both of which are essential biomolecules within all life-forms on Earth. The four bases used in DNA are adenine (A), cytosine (C), guanine (G) and thymine (T).  In RNA, the base uracil (U) takes the place of thymine (T).

Ribonucleic acid (RNA) is a molecule that is present in the majority of living organisms and viruses. Like DNA, it too is made up of nucleotides—which are ribose sugars attached to nitrogenous bases and phosphate groups. This is a nucleic acid is found in all living cells that have structural similarities to deoxyribonucleic acid (DNA).  

In Fig. 1 the makeup of RNA (left) and DNA (middle) present in most living organisms on our planet and referenced to those 4-ary (i.e., a group or set of four [quad/quadary] data elements—in this case the “nucleobases”), are used in the coding of DNA/RNA for storage.  The CG-content is depicted in the smaller diagram (far right).

From the theoretic information perspective, strands of DNA serve as a storage medium for 4-ary data over the alphabet {A, T, G, C}. The “alphabet” referenced here aligns with the four components of the DNA nucleobases mentioned in the biochemical descriptions above and as shown in the diagram of Fig. 1.  

Cold Data
DNA is part of the next-generation technology that can support storing mass “cold-data” (i.e., “archival”) or information which does not require regular or continuous access. Pools of synthetic DNA are being proposed as a potential medium aimed at archival (long term) storage purposes. Through the use of coding and data processing, errors are prevented during the biochemical processing proceeding the development of the DNA strands.  

For long term storage, all of the data sequences must contain limited runs of identical symbols and a balanced ratio (percentage) of A-to-T (Adenine to Thymine) and G-to-C (Guanine to Cytosine) nucleotides. These compositions are referred to as “constrained codes,” which are a class of nonlinear codes that by proper processing, eliminate a chosen set of “forbidden patterns” from the codeword sets.  

 Promises and Encumbrances
DNA data storage promises formidable information density, long-term durability, and ease of replicability. However, information in this intriguing storage technology might also become corrupted. Experiments have revealed that DNA sequences with long homopolymers and/or with lower guanine-cytosine (i.e., “GC-content,” see inset of Fig. 1) are notably more subject to errors upon or when moved into DNA-storage. 

Guanine-cytosine content is the percentage of nitrogenous bases in DNA or RNA molecules which are either guanine (G) or cytosine (C). A higher GC-content level indicates a higher melting temperature. GC-content should be in the 30%-80% range, with 50-55% being ideal; and the GC content influences the evolution of proteins because of energy cost. A brief summary comparison of RNA vs. DNA are shown in Fig 2 (noting the CG-content in blue).

One probably never realized the complexities in storage beyond how magnetic areal density or how the fluctuation of magnetic energy relates to the pickup head of a spinning magnetic (hard) disk drive; but storage density will always need to increase if we’re going to sustain this ever-growing thirst for digital data. 

When or if DNA data storage becomes prominent is still a vision into the future, but it is being promoted by companies including American companies Illumina, Microsoft, Iridia, Twist Bioscience, Catalog and Thermo Fisher Scientific. According to Markets and Markets, the DNA data storage market is projected to grow from $76 million in 2024 to $3.3 billion—growing at a CAGR of 87.7%—by 2030. 

In looking for the most intriguing topics to address related to cloud for 2023, I began by exploring what were the most significant challenges for cloud technologies and found that one of the most repetitively stated challenges related to “security” and maintaining “data integrity” (Fig. 1).  Data breaches remain one of the most significant threats facing cloud computing today. 

What did I find in my search? Most reports predicted that cybercriminals would continue to target the cloud as a means of gaining access to sensitive information. Summarily, the kinds of sensitive information included customer data, financial records and proprietary business intelligence.

Figuratively, most organizations today operate to some degree in the cloud. While employing the cloud simplifies operations in many ways, this comes with its own set of risks that can significantly impact the bottom line for enterprise and similar scaled organizations. From a report published by Lookout, an IBM Security Cost of a Data Breach Report (prepared in 2021), found that “the average cost of a public cloud breach was $4.8 million.” 

The Challenges 
A significant grouping of priorities related to IT initiatives now involve cloud services and tools, automation and DevOps—which continually evolve as leaders seek to unlock new efficiencies from the front office to the back and every space between. The findings of a CyberArk report recently issued suggested that this technology adoption rate will see a 2.4x growth in human and machine resources, which is coupled with a 68% increase in the deployment of SaaS tools for such services.

This surely means that utilizing the cloud for operational activities is essential, as when trying to build out the scale of similar services on-prem (including construction, supporting and managing) is found to be many times more costly. Furthermore, of the many elements incorporated into developing a SaaS environment or their application is that of creating a set of “secure” authentication steps.

Creating identities that can authenticate the human user(s) and/or the machine(s) involved, can be automated in the cloud, which will significantly reduce the hands-on requirements otherwise required for upkeep, deployment and maintenance. The growing number of SaaS activities businesses must address in the digital future will be highly dependent upon these evolving cloud services.

What Can We Expect?
Trending technologies in 2023 included the Internet of Things (IoT), blockchain, artificial intelligence, machine learning, Kubernetes and docker. With many of those technologies already in place and in full use, we can expect other new technologies such as quantum computing, cloud gaming, augmented and virtual reality coming forth in the near term/upcoming years.

What will cloud computing be like in 2024? Expect a nonstop evolution of new capabilities enhanced by consumer growth, automation, virtuality and more.

Despite these advances, the top challenges expected in cloud computing seem to remain almost the same as they were in previous near-term years (i.e., that last three to five years). We distinguish cloud computing as characterized by those processes and components associated with “deploying computing services,” such as servers, storage, software, analytics, databases, networking and intelligence. Such services rely upon deployment, and of operations over the internet, which characteristically offers flexible resources, faster innovation and economies of scale.

Data Security and Privacy 
At the top of the challenges chart (Fig. 2) continues to be that of data security and privacy (including customer trust). 

Not unexpected in this group is the challenge of password security and protection. Try as we might with multifactor authentication (MFA), people still don’t fully understand or recognize the importance of having a secure, unique and protected password. A 14-character, mixed alpha+numeric+special-character password is essential when working within any compute environment, including the cloud. Continually changing your password—while time-consuming—is an effective (and essential) part of maintaining that security.

We note that not all cloud providers can assure 100% data privacy, so users should understand the values in privacy and security protection (see Fig. 3 for validation). Another methodology to protect your data privacy is to routinely install and implement the latest software updates, especially on the network hardware and configure those components properly and fully.

Cybersecurity compliance includes certain compliance processes and ensures that the provider(s) meet industry standards, regulations, legislation—including international policies and procedures. The NIST Cybersecurity Framework and ISO 27001 are both excellent guidelines for the prevention of cyberattacks and compliance. Even if you don’t believe you’ll be “working” internationally, you should still follow such guidelines as data may indeed cross over to those parts of the world without you knowing it.

Multicloud Environments
Given the growing number of cloud service providers, users will be expecting to work amongst more than one cloud platform, even sometimes to support the same applications or activity. A “multiple public cloud services environment” includes services provided from different vendors within one architecture at the same time. For instance, a business might use AWS for data storage, Google Cloud Platform for development and testing, and then Microsoft Azure for disaster recovery.

We also hear the term “multicloud computing.” Fundamentally, there are three main types of cloud computing: public cloud, private cloud and hybrid cloud. Today, using one or more of these is not uncommon. (I discussed multimedia cloud and hybrid cloud uses and values in my October 2021 column, “Evolution of Multimedia Cloud;” my February 2022 column, “Cloud Production for Media,” and December 2023 column on “Hybrid Cloud Choices”.)  

A private cloud is one built, usually by the owner, for its own independent uses and it most likely would be built on-prem. Public clouds have the most familiar and recognizable cloud service naming with provisions from Google Cloud, Microsoft Azure, Amazon Web Services (AWS), IBM Cloud, Oracle Cloud Infrastructure (OCI) and others. 

Each of these public offerings differs in varying ways and can offer hybrid cloud services and migration paths from one platform to another. Be sure to crosscheck the capabilities from each vendor’s offerings when developing a cloud architecture for your uses.

Performance, Reliability and Availability
Interoperability, flexibility and performance are another set of challenges but possibly less expected are the performance and reliability/availability of the services to, from and within the cloud. Transferring large data sets (volumes) between cloud data servers depends upon sufficient internet bandwidth, which is a common problem. 

On the topic of availability—as with any internet service provider—getting to (or from) the host is usually a core “bottleneck” concern that is essentially out of the user’s control. And of course, once “in the (public) cloud” a user is now in a somewhat “hands off” world where the internal architecture of the cloud is something that you can only minimally affect—and are often determined by the SLAs written into the cloud agreement.

What are the Drawbacks?
A more serious and probably obvious challenge will be the lack of knowledge. Finding the appropriate cloud talent is another common challenge when maneuvering the cloud computing environment. 

As workloads increase through cloud dependencies, so do the number of tools available to global users. Enterprises, regardless of size, need strong expertise in order to properly utilize a growing set of tools and capabilities in the cloud. The solution here is to use/hire cloud professionals who have DevOps and automation specializations and experience.

Karl Paulsen is a frequent TV Tech contributor who has been writing about storage, the cloud and media solution technologies for the past three decades. He can be reached at karl@ivideoserver.tv.

Cloud and IT go hand in hand, so as IT leaders, you likely need a comprehensive, clear insight into the technologies that feed both the enterprise and the cloud. Of paramount importance to IT leaders and cloud architects is keeping your technology assets secure, well-governed and cost effective. This is a far cry from where IT was a decade or more ago, whereby IT was pictured more as support for the back office and keeping “the network” functional—as well as supporting the users and workplace.

IT leaders need comprehensive, clear insights into their technology to fuel this evasive and evolving data-driven, decision-making processes, which lead to the best of results. The cloud, while convenient and less involved (compared to an enterprise-size data center) is not without its concerns. This is not to be “negative” about the cloud. Quite the contrary, knowing pitfalls of the cloud can only make your implementation(s) better and less risky.

By looking at known issues, we hope to broaden perspective and set the tone for understanding how, why or why not “in the cloud.”

Lifecycle 
Most products sold to users have some concept of how long it will last. This “lifecyle” is no different for automobiles, appliances and certainly electronics. Like hardware, software also has its own version of how long it will last—sometimes based on the hardware it lives on and sometimes just how long the original equipment manufacturer (OEM) wishes to support it or feels it is worthy of maintaining due to technology changes (usually advancements) or their cost of keeping it alive. 

In IT (and cloud) , there is also a software lifecycle. In this case, the software asset lifecycle is about having your IT resources accounted for, cost-effective and properly employed. Vulnerabilities of the products lead the list of concerns about IT assets according to Flexera’s 2021 “State of IT Visibility Report.” This issue is a chief concern of the enterprise information management and staff. The same issues can be and are extended into cloud practices and, like the “ground-based” enterprise, must be carefully understood, watched and protected. The sidebar provides simple definitions as they apply to software systems and hardware components.

In considering cloud, many look at using a combination of ground-based (on-prem) services and cloud services. This is generally referred to as a “hybrid” cloud service and some feel this is not only important, but it may also be essential to its operation. In this model there is a dual role—that is, here the enterprise must manage its own services (on-prem) and in turn manage its cloud services as well.

Is a Hybrid Cloud the Right Answer?
According to Morpheus’ “Gartner Market Guide for Cloud Management Tooling,” “The requirement to support hybrid and/or multicloud deployments is stressing current enterprise operational processes and tooling that had been designed for their on-premises environment. The main use cases continue to be around cloud governance and resource management as enterprises try to avoid overspending or falling prey to security breaches.”

By definition, a hybrid cloud is a mixed computing environment where applications are run using a combination of computing, storage and services in different environments—public clouds and private clouds, including on-premises data centers or “edge” locations. 

On a broader perspective, hybrid cloud architectures are widespread primarily because almost no one today relies entirely on a single public cloud. Figs. 1 and 2 show the values and benefits of employing a hybrid cloud to your solutions.

The value is that in hybrid cloud solutions you need to migrate and manage workloads between various cloud (and ground) environments. In turn, this allows users to create more versatile setups based on specific business needs. Many organizations choose to adopt hybrid cloud platforms to reduce costs, minimize risk and extend their existing capabilities to support digital transformation efforts. 

A hybrid cloud approach is one of the most common infrastructure architectures of modern computing applications for IT, media, healthcare, and the list goes on. Today, most cloud migrations often lead to hybrid cloud implementations as organizations often have to transition applications and data slowly and systematically. Hybrid cloud environments allow you to continue using on-premises services while taking advantage of the flexible options for storing and accessing data and applications offered by public cloud providers, such as Google Cloud.

The Good and the Bad
On the other side of the coin, there are many reasons why there are plenty of workloads that will never go to public cloud—some of which include: regulatory response, life/safety reasons, subscription vs. permanent cost models, less control over your data security, and of course, you must have good Internet. 

The “if it’s not broke don’t fix it,” and the “one throat to choke (your own),” along with long-term costs of cloud computing being higher, all stack up against the “all in the cloud” model. Users also say some applications actually run better on a local server along with not knowing “where” your data really is (risk of regulations that could impact data retention or recovery, and how much does it take of your time plus the inability to control reliability.

Another not-so-pleasant concern is that every action leaves a trail (i.e., Where’s your privacy?). Many of us grew up thinking that almost everything online was anonymous. Wrong. Connecting to the web generates an IP address. Every website we visit can see that IP address, and others can “see” that information, everything from what operating system we use to the size of our screen resolution.

Nothing you do is private any longer, especially when your interaction requires or expects one to “create an account.” The main point in creating an account is to retain certain data in order to display it again later. (Privacy is lost, even if the website “says” differently. If it weren’t important, why does that site need it?)

Everyone Has a Data Profile
Every detail about us can, and often is, regularly bought and sold; cookies track us routinely. Even if you don’t accept the cookies, there are means to track and trace you. Artificial intelligence now “fills in the gaps” using other resources, such as Facebook, one of the largest repositories of personal information on the planet.

This issue isn’t limited to services with public data (as in Facebook and Twitter/X). Amazon, Google and Dropbox each store different but very intimate details about each of us. Personally Identifiable Information (“PII”) is valuable to you, to businesses and to hackers. Personable data generates a “profile” that now positions you into “classes” or groupings, which now link to other connections and end up being “mined” by organizations that profit from knowing what they know and, at times, exploiting that information for less than personal reasons.

And it’s not just cybercriminals who want your data; countless services and governmental agencies want and collect your personal details too.

HR Concerns
So, think further about your own organization or enterprise data, much of which may indeed include “your” personal data. For good reasons, HR departments take worthy concerns over their employees’ personal information and that you, the employee, trust that data to your employer. 

The same might go for corporate records, contracts and such. Hence the interest and concern over security whereby they may employ technologies like blockchain to protect transactions, contracts and other confidential information.

Types of PPI
Direct identifiers, also called “sensitive PII,” are data sets that can be used to pinpoint you and only you. Quasi-identifiers, also called “non-sensitive,” are those details that can be combined with other quasi-identifiers to “label” you—classify or group you for geographic, domestic or other reasons. Quasi-identifier designations are often used for statistical analysis placing you into group(s) that describe which “you belong to.”

Direct identifiers, as mentioned, are about you and only you. Nothing you share with another person is a direct identifier—that data such as your full name, medical history, credit card details, personal identifiers such as social security or insurance numbers, or your passport number. 

Europe took a hard stand less than a decade ago with its General Data Protection Regulation (GDRP), a European law enacted in about mid-2018). The GDPR concept aimed to protect you (the user) and provided a legal means to have to prove that any collector of said data had actually scrubbed and/or expunged your data from their files, should you request such action. 

Initially, the 1995 EU Data Protection Directive set goals and requirements, which the EU member states were free to interpret its general goals and requirements as they see fit when complying with their national (EU) laws. Essentially, this says that organizations must have a lawful reason for collecting personal data; the amount of personal data collected must be limited to the minimum necessary to complete the lawful purpose; and that data must be deleted once the lawful purpose has been completed. 

The U.S. (country-wide) has been struggling to develop a similar protection, but has essentially gotten no-where; yet California has implemented similar policies.

Where is my Data, Really?
The depth of these kinds of actions apply not only to local servers and services, but extend into the cloud, which leads to an interesting caveat. What happens when the data actually resides in a cloud that is in a non-EU country?

Hence, you can see the concerns for using “the cloud” when you don’t know where the data is or under which jurisdiction that data might be controlled at any given moment. Furthermore, these concerns become more complicated as the cloud providers expand and the resiliency models (data duplication and distribution) grow for faster, more improved services.

So, know your solution set thoroughly before jumping on the “all in the cloud” bandwagon. Keep informed or use a knowledgeable entity to help support and maintain your investments.

In recent years media organizations have been preparing for changes in how media is produced, which inevitably came to fruition once the impacts of Covid were realized. Prior to the pandemic, remote production was quite stable for those larger venue-like productions (concerts, sports, entertainers), however day-to-day operations (news, weather, local shows) had to be re-envisioned on a much broader scale once production crews became isolated.

The media and entertainment industry saw a huge impact as the industry nearly shut down overnight. Once organizations were able to survey the availability of their own resources, the extension of those resources quickly expanded, supported by software-enabled internet-based interconnects and flexible remote applications that were portable and adjustable.  

‘Remote Mode’
When the pandemic was “officially” over, the entities that enabled these new routine remote operations quickly understood that they could continue in the “remote-mode” fashion and sought to keep that new operational model functional. This obviously pleased the majority of those who had now resolved to work from home and could easily and effectively operate in that model with only marginal impact on homebase facilities. 

Certainly, exceptions popped up and adjustments had to be made—but the overall design and modeling for remote production was established. The foundation for many live and non-live productions were now setting like wet concrete.

Other things besides connectivity and applications have been changed or augmented to allow remote or at-home production to be sustained. The most obvious is the cloud—but cloud facilities had to expand in similar fashion to allow home-based central equipment rooms of the broadcast facility to be more easily utilized. 

By that I mean, accessibility, bandwidth, networking and fast, easily accessible storage all had to be reshaped in order to support live and non-live production requirements.

One of those areas, storage growth, has not slowed down in the least. Predictions say that by 2026 more than 220 exabytes of data (equal to 220 quintillion (1018) bytes) will have been created and stored somewhere in various ecosystems, according to research and a report by Coughlin Associates.  

The Varieties of Cloud
Reports have indicated that server and storage spending are leading indicators of the global economy. As for the future, companies are either terrified or optimistic of that pattern, or both at the same time, as predicted by NextPlatform in September 2022. IDC reported that total revenues in “shared cloud,” “dedicated cloud” and “all-cloud plus non-cloud” reached $39.9B; and overall increases from 2020 for the same period reached over $10B in growth. 

This only covers a single segment of those overall applications, and when compared with past sales—including service providers and everyone else (private users, non-shared services, etc.)—total revenue in 2020 was $130.9B and the predicted revenue in 2026 will reach $197.3B (references from IDC in 2022 as a CAGR of 10.9% from 2022 to 2026). 

With this growth, how will storage technology be able to keep up with the supply and demand expectations? What is changing to meet these goals? There is a steady change in storage technology platforms, which are necessary to meet the objectives expected. Some manufacturers are betting on the change to an all-flash storage device, whether for the local device or the cloud. One manufacturer recently reported that they expect capacity of their proprietary DFMs (direct flash modules) to increase by six-fold in a few years, reaching up to 300 TB capacities.

Advancements in 3D NAND (Fig. 1) areal density will look somewhat like how HDD (hard disk drives) grew a few short years ago. The physical capacities of such hard drives nearly reaching saturation as molecular densities drove magnetics to practically crashing and colliding upon themselves. Today, 24 TB and 48 TB DFM drives are shipping, dwarfing past other devices and showing that HDDs may no longer be the norm. 

How is this changing? Heretofore, 3D NAND devices used a stacking or laying principle with between 112 and 160 layers per IC. In the next few years (and less than five), fab vendors expect the number of active layers to increase to between 400 and 500 layers, yielding much higher-capacity 3D NAND ICs. 

Other Storage Segments
Applications for storage and its growth indicate that there are other storage segments which also need to be supported by M&E.

Long-term archiving will continue on various media form factors, with external HDD deriving around 18% of the archive storage space and 20% for local storage networks (NAS and SAN) with about an equal amount for private or public cloud at 21%. Digital magnetic tape (e.g., LTO or similar) remains the leader at 32% overall, according to Coughlin Associates’ research—but this may be shifting downward as cloud becomes more convenient and affordable. This prediction, again, is for long-term storage and for short-term storage the model changes even more dramatically. 

Both bandwidth and storage are required for a proper balance in content generation (i.e., production, capture, post and readying for transmission). The evolution of the network to 100 Gbps is almost routine now. Effectively, new installations striving towards an all-digital post-production environment should consider no less than 100 Gbps pipes to address the up-and-coming virtual environments that already require 20 to 50 Gbps connectivity/communications on a consistent basis.

Multipliers for content capture command more and faster storage solutions that can only be met by adding storage (local and/or cloud) and pipes that move data at rates approaching 50+ Gbps continuously. If you’re producing 4K (or 8K for long-term archiving) the network must be capable of handling 100 Gbps with the servers and storage able to receive (and return) such data rates. Today, for most storage systems in the M&E space, the bulk of those facilities continue to rely on HDD, however, SSDs are playing an increasingly important role through the use of NVMe SSD and NVMe over fabrics leveraging the “pooling” of storage.

Storage pools (Fig. 2) are principal constructs around storage virtualization. The “distribution” of a storage pool (a block of storage) may be connected in any number of means including Fibre Channel, iSCSI or direct attached (DAS) storage devices. The pool should not be “vendor-locking,” meaning that a pool should not be dependent upon one storage vendor’s product or proprietary solution—and instead allow flexibility to add or remove storage devices across the overall storage network solution. 

This, in turn maximizes the opportunity to scale or shift storage depending on the load or application; very important when exchanging files or resolution densities in a production environment.

One can see that the dynamics associated with these new and future production storage sets will require some careful planning to mitigate being trapped into a single source solution. This appears, on the surface, to be the target for future storage product providers, whether in the cloud or on-prem.

The magic just gets more involved. What “magic” you say? And one answers “all the magic”... which infers that the “magic” is still being defined, developed, and experimented with.  

As the title of this article infers, this “magic” consists of elements in Web 3.0 (which claims to empower creatives and users—both—to share in the value they create). The values, summarized in the introduction to Shelly Palmer’s Metacademy, provide businesses (whether new or existing) the knowledge and linkage into communities associated with these new entities including blockchain, NFT, and the metaverse. 

So how does this fit into your needs or interests in cloud-based activities?  A good example is blockchain, which is like “information carved into stone.” Such information (e.g., a contract or copy protected media) is configured as permanent, unchangeable, and ambiguous as to what you’ve written that data into.  Cloud-centric services can properly handle such configurations because the data lives on a distributed network across a large number of computer-based systems (servers) as in a mesh or a peer-to-peer (P2P) environment. 

Because the information is essentially “permanent” one can easily tell if anything associated with that information has been altered. Although you can “add” to the blockchain, you cannot modify the existing data set—thus rendering a protected value that can be recognized for its authenticity for eternity.  

Immutable Cloud via Distributed Data
Blockchain uses a cryptographic hash algorithm as a one-way function which is derived from the original data. Altering that data (even a date-code or GPS coordinate) generates a new unique functional representation, which can be compared against the original authenticated data and signifies that the original data and the current data are different.  

This original data is “hard-and-fast”—otherwise called “immutable”—which means it is unchanged over time and is unable to be changed. Thus, blockchain, in this described context, is protected and cannot be modified. Blockchain is extensible across any cloud service as an inalterable contract that is safe and valid irrespective of where the data is stored or sent to/from.

Sometimes called a distributed ledger, blockchain uses a digital file which is continuously growing via an encrypted transaction (the “blocks”) which are copied to a P2P network of distributed computers. Here, each computer (node) accesses information on a computer that serves information to its clients (i.e., other servers). Nodes on this P2P network create a consensus, i.e., an algorithm which is used to resolve conflicts and to ensure the accuracy of the blockchain. 

Distributed and decentralized ledgers are what power the system. P2P networks are very stable because the information is replicated in a multitude of places, irrespective of the scale of the system. This makes such applications perfect for a cloud-based environment, since the cloud (in theory) can function at any scale.

Hashing as Sequential Records
As the name implies, blockchain is a set of records that are linked—or chained—together sequentially using an index, a series of timestamps, a listing of each of the transactions, a “proof” and the hash (algorithmic set of mathematically derived data) from the previous block.  Alteration of any one block changes the hash of all downstream successive blocks which are unrecoverable or reversable.

A hashing function (Fig. 1) simply takes a variable number of characters (the ”message”) and converts it into a string with a fixed number of characters—referred to as the “hash value.”

Cloud Computing and Blockchain
The blockchain is a set of unchangeable and decentralized data sets (the blocks) which utilize a shared database. In cloud computing the system is orchestrated for delivering computing services.  

It is comprised of servers, databases, storage, and associated activities. To increase data security, cloud computing will use blockchain peculiarities. Due to its scaling capabilities, the cloud can provide on-demand computing resources for blockchain operations.

Secure, Unique, Adaptive and Semantic
Security is often thought of as “the elephant in the room” with most people having only limited familiarity with it. The goal of the new “Prime Web 3.0” is to change that perspective with examples which include ubiquity, decentralization, artificial intelligence, blockchain-security, and connectivity.

Web 3.0 is meant to be a bottom-up design with decentralization, open to everyone, and built on top of blockchain technologies and developments in the Semantic Web (Fig. 2), which describes the web as a network of “meaningfully linked data.” 

The World Wide Web Consortium’s (W3C’s) vision of the Web 3.0 is conceptually about linked data. The Semantic Web employs technologies that enable the creation of data stores on the Web, to build vocabularies and to manage or write rules for handling such data. 

Web 3.0 is empowered by technologies such as RDF, SPARQL, OWL, and SKOS, which has linked data within an internet framework that employs understandable machine-readable data. Web 3.0 is machine-interpretable, structured and interlinked, with open access sets of data repositories utilizing globally edited adaptive information resources composed of unique web resource identifiers for every bit of information. The concept is a bit like a media-based GUID (Globally Unique IDentifier) whereby each table data cell of a table has a unique identifier.

The current World Wide Web is more than sufficient, but as a tool, many feel they need much more from “the Web” (i.e., the original Web 1.0 and today as Web 2.0). The term “Semantic Web” is an extension to the current Web proposed a decade ago by the “father of the web” Tim Berners Lee, known as TimBL, who conceived and developed it in the late ‘80’s while working at CERN.  

Why is the Semantic Web Worth Having?
The current Web of linked documents can present a tremendous amount of human-understandable Web pages from a single search through lexical marching of the search keywords with similar or exact words found in the documents available on the web. 

This information is vast and mostly irrelevant to the user (especially after the first pages of the search result). Moreover, users have to scan through individual search results to make meaning out of its content. Such machines store and present the web documents knowing nothing about their content.  

On the other hand, a Web that can store data in documents that are both human as well as machine readable opens up a whole new experience of human-machine interaction. Allowing computers to understand Web 3.0 content allows one to only submit basic data or answer a single query before an entire transaction is carried out efficiently by a machine.  

Shared Digital Reality
Contrary to the decentralized Web 3.0 and blockchain concepts (the technology behind bitcoin transactions), the rather new and still somewhat confusing term, “the metaverse” is a shared digital reality enabling users to connect with each other, build economies and interact in real time. The metaverse doesn’t care who owns the data or its information.

Web 3.0 brings the internet into the future—noting that we’re still in Web 2.0, which was developed two decades ago and emphasizes user-generated content, collaboration, and social networking. Web 3.0 further allows users to “manage and claim ownership of their works, online material, online personas and digital assets.” 

Currently, many businesses only concentrate on the development and delivery of their goods and services. So the evolving Web 3.0 evolution tends to overcome some of the drawbacks or flaws in the previous Web 2.0 internet era by addressing and administering crucial concerns including data ownership, control and management.  

The “metaverse is a global 3D network of virtual reality worlds” according to some and further to that “the metaverse is a hypothetical version of the internet” which is confined to a uniform, global virtual world using VR and AR headsets in a near science fiction, futuristic working environment. 

Examples include the use of avatars as personal representatives, interchange costs or sales through digital currency exchanges, and complete purchases or acquisitions in its own in-world currency. Users thereby travel aimlessly through the “metaverse” in a means not heretofore used or found practical.

Browsing (the internet) now becomes navigation of a virtual world which mirrors parts of the actual (“real”) world now known affectionately as “the metaverse.” Thus, users now create and enable a new virtual economy, without the transactional limitations of (previously) modern living conditions.

Not to be Confused
Here is where the actuality vs. practicality and the goals of the Web 3.0 experience seem to collide. Gamers understand this new reality—they embed themselves in these interactions which value immersive, interactive, and social platforms to extend their VR/AR/AI experiences without boundaries.

Web 3.0 users and its community have decentralized (i.e., unconnected) ownership and control over the Web. The metaverse, on the contrary, is a shared, digital environment focused on the ability for people to communicate, create their own economy (i.e. the use of bitcoin) and engage in real-time interaction without ownership concerns or knowledge of the “real” owner.

The cloud is essential for this to occur since it knows no boundaries and serves no limitations. It may indeed take decades to see the many wonderful values of the metaverse and its intrinsic differentiations in the evolving Web 3.0. Advancing, aggressive technologies are critical to these foundations… so watch this evolution carefully—they are not easy to fully comprehend, but certainly command an unexpected eye-opening set of changes in our world. 

Cloud architectures may include a few definitions that surround the overall concept of what the “cloud” is as well as “when” or “how” it began. Some have conveyed the expression that “the cloud is really only somebody else’s computer,” which may be a bit shy of reality and more of a tongue in cheek colloquialism. According to the Journal of Accountancy and IBM, the concept of “cloud computing” first arose in the 1950s in the form of dumb terminals connecting to a mainframe computer.

We hear more of the term “cloud computing,” which is mostly related to cloud as a service, and not an architecture. The term “cloud computing” was coined within a 1996 Compaq Computer Corp.’s internal document (by George Favaloro), with the term “cloud” originally linked to the concept of “distributed computing.” In May 1997, Sean O’Sullivan of NetCentric, attempted to trademark the phrase “cloud computing,” but abandoned that effort in April 1999. The term went mainstream at Apple-spawned software and electronics company General Magic in the early 1990s, with even earlier mentions in academic work before that.

The ”where” of the term is purported to be when the British computer scientist Christopher Strachey (National Research Development Corp., London) published an academic paper at the International Conference on Information Processing, which was also about the shared use of mainframes, called “Time Sharing in Large Fast Computers” (1959-06-15) available for download at archive.org. The paper covered several of the initial issues that had to be addressed in the very early stages of compute processing and operations.

When it comes to the origin of cloud computing, the public generally believes that Eric Schmidt, the former CEO of Google, was the first proponent of the concept when on August 9, 2006, at the Search Engine Conference (SES San Jose 2006), he proposed the concept of “Cloud Computing.” In its early stages, circa late 1990s, the cloud was used to express “the empty space between the end user and the provider,” per a brief “History of Cloud Computing” by Keith D. (Foote 2021-12-17). Additionally, Professor Ramnath Chellapa of Emory University defined cloud computing in 1997 as the new “computing paradigm, where the boundaries of computing will be determined by economic rationale, rather than technical limits alone.”

Not Fully Defined
As mentioned earlier, in its simplest of terms, the cloud (operations-wise) is truly the “use of someone else’s computer.” As the evolution evolved into the early 2000s, five key cloud characteristics emerged: on-demand self-service, broad network access, resource pooling, rapid elasticity and measured service. Industry-recognized companies and other authors have also concluded that “a [cloud] solution must exhibit these five characteristics to be considered a true cloud solution.” 

This doesn’t mean it must possess all five characteristics, however, some or more of these topics are anticipated in some structure to represent cloud and cloud computing. A graphic depiction of services in a cloud computing environment is shown in Fig. 1.

Cloud Deployment Models 
We have further established that there are four familiar types of cloud deployment models: public, private, community and hybrid; each quite similar but separated by who or what entity provides those services. NIST also defines these four cloud deployment models according to where the infrastructure for the deployment resides and who has control over that infrastructure. 

Deciding which deployment model you will engage is one of the most important cloud deployment decisions you will make. Cloud deployment models satisfy different organizational needs, so it’s important that you choose the right model to satisfy the needs of your organization. 

Most important is that each cloud deployment model has a different value proposition and different costs associated with it. In many cases, the choice of a cloud deployment model may simply come down to economics. 

Private Popularity 
Although initiated in the 2008-timeframe, “private clouds” were still undeveloped and not very popular—but public opinion began to change once companies like AWS and Microsoft solved many of the public’s security concerns. 

Hybrid clouds began to pop up around 2011—a concept that required an interoperability model be established, which allowed private and public clouds to shift workloads between two (or more) clouds. Business systems lacked the ability to accomplish this until the now ubiquitous three structures—created in part and recognized by Oracle Cloud in 2012—created Software as a Service (Saas), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) foundations that nearly all clouds employ for their functionality in part or in whole. 

Multi-clouds arose once organizations began as SaaS-developed services such as HR and CRMs, and supply chain management became optimized for cloud services around mid-2010–2020. Multi-cloud let users select the best-of-breed cloud to perform specific services while allowing those independent clouds the ability to interchange data and operational solutions among each other.

Developer-Driven 
Cloud began to shift from developer-friendly to developer-driven practices around the 2016-time frame. Application developers started to take realistic advantage of the cloud because of the tools the cloud had available. 

Services strive to be developer-friendly to draw more customers. Realizing the need, and the potential for profit, cloud vendors developed (and continue to develop) the tools apps developers want and need. Practices that include containers (e.g., Docker, Kubernetes and other open-source capabilities) are driving the resource-needs capabilities, which all clouds—irrespective of the structures—are moving toward.

Applications and Resources
Many are familiar with cloud-computing resources, yet it sometimes comes with the uncertain knowledge of “what” those resources are. Essentially, cloud computing is “a method of delivering computing resources.” First, those evolving cloud-computing services supported the needs for data storage and processing, and eventually moved on to software. Cloud-computing business systems, such as Customer Relationship Management (CRM), have become available instantly and on demand—changing the dimensions of those services traditionally dedicated to mainframe-like computing or application-specific packages provided by Oracle and others in their heyday. 

These low-cost-of-ownership models for cloud computing have gotten lots of attention and are seeing increasing global investment. In times of financial and economic hardship, this approach becomes more affordable on a OpEx (pay-as-you-go) model instead of up-front investing in hardware (CapEx) and fixed-priced applications that strain budgets and become valueless as technology capabilities increase.

From a generalized perspective, cloud computing provides users with implementation agility, lower capital expenditure, location independence, resource pooling, broad network access, reliability, scalability, elasticity and ease of maintenance. Each of these topics will play differently into your business needs. And similar perspectives can be applied to entertainment-based media from an operations (or OpEx) model, but will generally be applied with differing practices and payloads (i.e., media content and not monetary hardware-centric considerations). 

Legacy IT infrastructures can no longer provide those needed services that will allow the organization to remain competitive. Continuing to add expensive hardware in order to maintain massive IT-centric data centers are gradually, if not assuredly, coming to an end. As evidence of this trend, according to IDC, worldwide spending on public cloud services grew 26% in 2019 to a total of $233.4 billion, up from $185.2 billion in 2018 (Oracle’s “IaaS for Dummies,” 5th edition).

Those users who are ready to migrate services to the cloud need to understand and even directly ask their potential cloud service providers some important questions.

Know Your Needs
Fundamentally one needs to understand which types of applications your potential provider would be set up to run. Know and itemize what your (the user’s) needs will be. Are those needs enterprise applications (such as ERP, supply chain management or human resources)? 

Or is the potential cloud provider a technical compute-centric solution provider (such as high-performance computing—HPC or data and/or database analytics). Perhaps your organization’s needs are focused on marketing or web-scaled applications such as ecommerce or mobile? Do you have an explicit need to support social networking or video streaming and delivery services?

Cloud-to-Cloud or Single Source
Be sure you know the nature and best approaches on how the cloud providers’ services match your needs. You may find that more than a single service provider will be required, thus you need to know if there are means to interact with more than one cloud provider—that is “cloud-to-cloud” services.

If you’re in the media and entertainment industry and you plan to provide live cloud production services, can the actual applications you need or will enable be usable in that cloud? What are those costs to implement the ground-to-cloud solutions initially? And what will be the expected usage periods for cost of operations?

Not Always Equal
Noting further that some providers may indeed suggest that their cloud is engineered to support every application, which will, in turn, require a bit more investigation on your part to assure that your needs are best suited in their cloud vs. another’s cloud. Remember, not all clouds are created equal. Fig. 1 provides examples found today in many of the cloud provider services, some of which may be useful in live M&E applications; and other services may need to be created to support the specifics of certain use cases—including live “production in the cloud.”

The next major decision is to investigate the effort, costs and requirements to migrate your applications to the cloud. Again, looking at the M&E model for live production, know what is required to move the live studio or field content into the cloud, including minimum and maximum bandwidth needed, the content compression expectations, and especially the overall set of latencies that will be expected. A live music video concert will have completely different requirements compared to a remote sporting venue or a talking head interview with guests that are half a globe away from the primary cloud’s site. 

Scale, Performance and Control
Another consideration is the scale of the environment. Users or service providers should not be expected to rewrite your (or their) applications in order to fit their cloud agenda. Performance and control are equally important compared with or against the other objectives such as compression and latency. 

Specialized Adaptation
As cloud adoption increases, operating expenses for managing an organization’s IT components also will grow in complexity. This is similar, to a degree, for cloud implementation. Users should engage a cloud provider that is simple and straightforward to manage; minor changes, typical in live production applications, should be easy to alter or augment without having to engage a consultant or third party to make those adjustments. 

Do scripts and updates require a knowledgeable human who knows the cloud’s operating environment in detail or are changes allowed using a simplified set of menus? Furthermore, changes should be expressed in billable dollars—that is, what will it cost to make those changes and for how long will those costs be expected in terms of length of run time or upticks in computer processing costs?

Change is Expected
How automated are those changes going to be and how long does it take to “spin up” those changes? How will those changes impact the databases? Are these system-wide, autonomous or some other factor to functionality going to “upset the applecart” or run the risk of collapse and/or create a reduction in performance? How much “manual labor” will be needed to install, update, adjust or monitor the applications you expect or need to run?

Given the topics expressed earlier, does the primary cloud service provider offer a multi-cloud solution? Can and how are loads distributed should there be a crash or a need to connect across global regions? Is your cloud provider in a competitive mode or just a simple service provider mode? If multi-cloud operations appear to yield better performance capabilities, do both sections of the two cloud providers play nice together or will there be hurdles (costs, egress, performance) that must be overcome that might not yield a net-net benefit?

Cost, Security and Trust
Security is always of concern, whether on prem or in the cloud. Do the expected cloud provider’s security practices allow for harmonization, or do they conflict with each other? Can your potential provider be configured to mesh with corporate security practices? Is zero-trust a mandated practice within your organization and will the cloud provider’s practices align with those policies? Will the security practices be easy to accomplish, or will there need to be overlapping tools employed that could reduce performance, increase complexity or add latency to the live applications?

Costs are always inherently a part of the delivery equation. Pricing models are often confusing and, in some cases, will put additional layers of costs especially when moving data into the cloud and then extracting. Users should insist they fully understand and can predict the costs for all the services they need. 

For example, be sure that at the conclusion of a particular service experience that unnecessary services are terminated and shut off to preclude the continual “meter is still running” impact. But also understand the balance of any “startup” costs that would be incurred as the session starts up or is being configured. In other words, costs can creep in from any corner of the equation or operation—be certain these costs are identified, are manageable and can be mitigated during times when the services are not in active duty.

Contract Considerations
Extensible, contractual costs may help mitigate the larger expenditures, but understand those contract expectations before engaging. If your service does not pan out from a benefit perspective, be certain you can back out of the contract easily and without additional penalties. 

Also, since services are continually adjusted or added to for each cloud provider, you may find that, after six months of services, another new provider now offers similar services at less cost—and you should be ready or prepared to move providers if the overall savings are worth the effort.

Code and Efficiencies
Applications require code and developing the efficiency of that code is paramount to improving the performance of the cloud services employed. Having the right tools and design for the services are keys to ensuring sufficient profitability and use of cloud services. Leveraging the time, cost and resources are the pinnacle to success in a cloud service implementation. 

Like any three-legged triangle, each leg of this “stool” must support other remaining legs. Traditional practices used in the development of these new cloud-based applications will typically be too slow or too cumbersome when deployed as a “cloud-native” solution, so expect to spend time, money and resources in finding new means to deploy your applications and needs. 

Monitoring and continual analysis of the complete solution will become a new factor in the implementation of any enterprise-grade cloud solution. If your new potential service provider offers these tools, learn how and when to use them—they will be necessary to ensure that peak performance for least cost can be achieved and monitored. 

Cloud services (as shown in Fig. 2) mean users need to rethink the business and technology approaches to their business. When reaching out beyond the cloud provider’s traditional services, be certain you ask all the questions of each potential provider. Get the best answers you can and then be sure the contract(s) you engage in meet those expectations. 

Under the cloud umbrella are the familiar variations—public, private and hybrid clouds. The latter, hybrid, is an environment featuring a combination of both public cloud (e.g., AWS, Azure, Google, others) and private cloud. 

By “private cloud,” we mean a “physical location” such as an on-prem data center, a “co-lo” type center, or a managed service dealing directly with users on a contractual basis. Many variations in private clouds exist. Facebook, for example, created its own private cloud for its infrastructure and does not utilize a public cloud to store its data. When scale, security or functional need dictate something outside the boundaries of a public, commercial cloud, users may turn to this privately controlled model.

With those basics defined, an emerging user trend is now developing. Certain specialties now command specifics that demand sets of capabilities that span more than a single service whether public or private cloud. While such a model might be classified as a hybrid cloud environment, there are places and applications where that traditional hybrid cloud implementation expands outside those classic dimensions.

Hybrid Prevails
To place this into perspective—almost no one, today, relies exclusively on the public cloud for computing. Today, a hybrid cloud is categorized as a solution or environment in which applications are running in a combination of differing environments. Reasons include capabilities that cannot be supported exclusively in any public cloud—for example, the production of an entertainment program that demands services at a venue that can’t be done in a virtual cloud platform (live sports clearly needs cameras, sound and human interaction).

Hybrid models are here for the near term, yielding new opportunities to assemble other elements of the production using tools that heretofore were relegated mostly to large frame, custom hardware. Such development is going full speed, but it’s not clear when this will become mainstream. Thus, there are cloud service providers who are focusing on supporting such migrations using specialized capabilities (their own “secret sauce”) that set them apart from their competition. 

More than One Cloud
Enter a relatively newer approach to methodologies that promote multiple methods to reach the users’ goals. This relatively recent trend is referred to as “multi-cloud,” (Fig. 1)

Multi-cloud and hybrid cloud models refer to cloud deployments that integrate more than one cloud. Each differ in the kinds of cloud infrastructure they include (Fig. 2). For example, a hybrid cloud infrastructure blends two or more different types of clouds, while multi-cloud blends different clouds of similar type.

Multi-cloud, according to IBM, is “the use of cloud services from two or more vendors,” inferring two or more different public cloud vendors, yet that doesn’t necessarily mean only “public cloud” providers. The multi-cloud approach gives organizations increased flexibility to optimize performance, control costs, and in turn, leverage the “best of breed” cloud technologies available. 

Services Today 
To set the framework, we need to examine where the technologies lie today. The three better-recognized “cloud services” include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Each of these three services utilize some degree of “cloud computing,” usually contained within that cloud under specialties and services that the specific provider has optimized for their particular cloud infrastructure. 

Each of the three services’ platforms employ what we refer to as “cloud computing.” By fundamental definition, cloud computing are those technologies that “make the cloud work.” More expanded descriptions include “a remote data center, which contains the (computing) resources that support those applications” and include physical (and virtual) servers, development tools and networking. Each of these services are provided in a cloud-like environment, i.e., “native cloud.”

Some definitions will include “via the internet” and others will describe cloud computing using a “network of remote servers” that are “hosted on the internet” for functionality, which includes “the management, processing and storing of data.” These remote data centers may be managed by a third-party cloud service provider (CSP) who then charges as a subscription-based service with on-demand billing. To the end user/subscriber, they are likely unaware of which services are provided on which cloud model, i.e., private or public.

Compute Offerings 
Leading cloud providers, as well as cloud-solution providers (e.g., VMware), may participate in multi-cloud solutions for compute. Wrapped in the cloud compute structure is “infrastructure, development, data warehousing, cloud storage, disaster recovery/business continuity” and more. 

Cross cloud utilization is where this multi-cloud “best of breed” concept plays best. For example, artificial intelligence and machine learning may best be practiced in a cloud service environment specially tailored for these operations, given that the scale of servers and storage necessary for a large AI/ML deployment can be intensive. In this example model, deploying a multi-cloud model might involve data acquisition from one cloud provider and compute analytics that might best be served in another.

Benefits gained in this strategy may be realized in different ways by various organizations. Live event or studio production might best be served by vendor A while post production (transcoding, formatting, versioning) might be better served by vendor B. Allowing the user to choose cloud services from varying providers based on performance, function, costs, etc., is now a marketplace-driven agenda compared to past models where you went to a single provider for all those services wrapped into a package—and where you might find certain service capability limitations.

Risk, Vulnerability & Loss 
Other important rationale for the multi-cloud model include risk reduction, reduced vulnerability to losses in data, compromise, unplanned downtime or outages, and more. Additional operation strategies that can leverage multi-cloud include licensing, security, service-compatibility and alleviation of signing up with a single provider who can’t offer all the services under one umbrella. 

Unfortunately, as all these cloud models grow, user/administrators have been facing what is referred to as “shadow-IT,” i.e., “you can’t protect what you can’t see.” The explosion of unsanctioned applications, software and devices is weighing heavily on the CIOs who must develop policies based upon the knowns. By limiting the services to a single solution, you could risk pushing the limits, which you can’t see, touch or know about until you arrive there and find “you just can’t do that here.”

Multi-cloud uses are still evolving. Full feature sets will require additional support and conduits to or from the variations. Inside the media and entertainment technology domains, efforts to establish best current practices for elements like multi-cloud—which requires ground-to-cloud and cloud-to-ground (GCCG) parameters—are moving forward. Harmonization of how well one cloud provider’s services (e.g., software-defined networking for compressed signal production) versus another’s capabilities (e.g., in compilation and distribution) will allow users to experience opportunities in cloud-to-cloud services that can more easily react to the needs of the entire ecosystem.

Migrating from on-prem to cloud storage can drive the inexperienced user to new sets of knowledge well outside those they would encounter when managing in-house storage. The most apparent differentiators are that in-cloud storage users pay on a per-consumption basis—usually monthly or at some incremental time-based period—and by the type of store they send their data into. Both conditions are set by the agreement established with the cloud service provider (Fig. 1). Scrutinized or scrutiny 

Definition-wise, cloud storage is a service model whereby data in one or more locations is transmitted to and stored in a remote storage system. Cloud storage components are managed by the cloud storage provider. Responsibilities of the storage provider’s solution include maintenance, backup and availability to the user. The latter, availability, is often stipulated by agreement, sometimes called a contract or service level agreement (SLA).

Storage availability has two components: a time-to-retrieve (recover) data factor and a cost-to-store factor. Usually, these two are tied together. For example, if the user puts its data into deep storage and does not depend on that data for routine (daily) applications, then the cost is much lower per unit gigabyte than for sustained, readily accessible, low-latency/fast-recovery storage applications—as in editing, rendering or compositing. 

VIRTUALIZING INFRASTRUCTURE

When storage is based on an infrastructure—which includes accessible interfaces, near-instant elasticity and scalability, multitenancy and metered resources—the storage is usually cloud-based and “virtualized.” Meter resources, also known as pay-per-use, are those offered with potentially unlimited storage capacity resources. Commonly found in enterprise-grade IT environments, this application has moved from a flat-fee (cost-per-month) world to a pay-for-the-use fee structure.

A familiar structure for pay-per-use is Apple’s iTunes model, the “sample for free and pay for what you want to ‘own’” (so to speak). Obviously in cloud storage you will not “own” the physical platform that holds your data, but you will pay for what you use based upon its structure, endurance/availability, its accessibility and the length of time you utilize the storage space.

LOGICAL STORAGE POOLS

A cloud service provider will manage and maintain the data that was transferred to the cloud by the user. Data is usually distributed across disparate, commodity storage systems. Data storage topologies may be on-premises, in a third-party managed data center or in a public or private cloud.

In an on-prem environment, for various reasons, data may be structured in logical pools. In such a shared environment, storage pools are capacity aggregated and formed from various physical storage resources. Pools may vary in size, yield variable but conglomerate performance (IOPS), and provide collective improvements like cohesive management and aggregate data protection. Logical pools are usually provisioned by administrators via management interfaces. A cloud infrastructure generally utilizes this logical form of storage.

RAW AND COOKED—LAKES AND PUDDLES

Storage pools may be equated to a data lake, although there are differences between these two derivatives. The data lake is a storage repository holding a large and vast quantity of unprocessed raw data known as source or atomic data. A giant bit-bucket where data is pushed without specific organization is a form of data lake. Processed data is, although less often, referred to as “cooked” data.

Raw data may not necessarily be called “information” since there needs to be an abstraction or applicational use, accomplished through processing, elevating its worthiness and value from raw to informational purpose. A data puddle is a single-purpose data mart built on big data technology, which is essentially an extract from a data lake.

ON-DEMAND STORAGE

No one likes to pay for things that they don’t need or use. In the cloud, storage services are provided on demand with elastic (increasing and decreasing) capacity as needed, when needed. Opting for cloud storage eliminates the requirements to buy, manage, depreciate and maintain storage infrastructures that reside on-prem, (Fig. 2).

Cloud models have radically driven down the cost-per-gigabyte of storage. However, cloud storage providers, also known as a managed service provider (MSP), may add different sets of operating expenses vs. those in owned, on-prem storage. Added OpEx costs could make certain cloud-based technologies more expensive, depending upon whether or not the equation considers how or when the storage is used. Look for options when selecting and architecting your cloud storage solution. 

THOUGHTFUL ACCESS SHUFFLING

Cloud providers concoct many useful and appropriate names for their various services, utilities and architectures. As of 2021, Azure and AWS each offer 200+ different products and services. Without naming anything specific, the concept of shuffling data storage sets to appropriately price and utilize varying locations in the cloud is a topic growing in popularity.

Optimizing storage costs, without burdening the user, employs automated methodologies that might be termed “thoughtful.” Otherwise called tiered storage, such architectures are not new, especially for ground-based (on-prem) facilities. In non-cloud datacenters, the practice of using high-performance storage for editing, compositing or rendering, where accessibility and speed is essential, is commonplace. This is known as Tier 1 storage.

A large SAN or Fibre Channel data store can be a costly proposition. Not all data workflows need this capability, so less-needed data is typically pushed to a Tier 2 (mid-level) environment. Archive data, which is seldom used or set aside for protection or redundancy (longterm) is known as Tier 3, where it may live on tape or object storage or both. Some may also use Tier 3 as that single occurrence where data is pushed to the cloud (known as deep archive) knowing it won’t require fast retrieval anytime soon.

CUTTING COSTS BY INTELLIGENCE

Early in cloud storage history, moving data around to different data containers (buckets) was accomplished manually and by specific direction of the user. While there may have been cost advantages to deeper storage, the labor effort was not advantageous to those early cloud storage users.

Things have changed over time as varying new services, increasing volumes of data and acceptance (i.e., trust) of the cloud provider by the user continually improve.

Accomplished by learning elements of data usage patterns, unattended cloud storage migration between cloud storage tiers was introduced in the 2017–18 timeframe. By automatically moving data between cloud stores, users and the cloud providers gained new advantages. With simple recognition of access periods for data, cloud providers would shift dormant storage to a “deeper” level without ever having to contact the user/owner of that data.

Users typically authorize automatic migration when signing up or establishing a particular SLA. Actions might be via user interface or APIs associated to work orders or activities. Adding intelligence to this practice has evolved over time. Intelligent tiering gives cloud service providers opportunities to expand archive tiers to other levels, which in turn improves the cost structures accordingly.

APPLICABILITY

When engaging cloud storage services, users should look into automated options and evaluate, via cloud-provided models, the value of tiered and automated storage electives. For large projects extending over months to years, one set of guidelines might be appropriate. Smaller projects utilizing shared data sets across multiple production activities may yield different answers.

Experiment with the numbers, frequencies, volumes and applicability to a particular workflow before going down any cloud storage path. Don’t let the “I’m not paying attention to this” excuse be the reason for higher storage costs, which then provide no improved results.

Karl Paulsen is chief technology officer at Diversified and a frequent contributor to TV Tech in storage, IP and cloud technologies. Contact him at kpaulsen@diversifiedus.com. 

Why is a storage architecture an important component in the organization’s overall media composition and delivery platform? Simply stated, the storage architecture is the foundation that sets the prioritization of an organization’s data management, performance, metering and content protection strategy. The concept is relevant whether for transactional processes or unstructured media-centric content creation and delivery.

Storage management architectures are one of the more important areas that define both the ease of operations and the success of the operation from a delivery perspective. Additionally, unprotected or improperly structured architectures can end in disaster; and it is not unusual to find one or both of these elements in some of today’s content creation environments.

Storage technologies have steadily evolved over the past 30-plus years, as depicted in the accompanying evolution timeline in Fig. 1. Today, architectural styles for storage may be composed of solutions provided by storage service providers, storage component vendors and dozens of experienced (and some not so experienced) consultants or solutions providers. Given the system complexities, home-brewed storage at an enterprise level just isn’t very practical any longer. Selecting an appropriate style or foundation for your storage architecture becomes key to its performance and the continued success of your operations.

GETTING THAT CONTENT DELIVERED

As an example, in a broadcast television news organization the time to get a story “on the air” can make or break that program from an audience attention span and ratings perspective. Everything in a “breaking news” headline is based on who gets to see that story first. Coupled with a story’s promotion is the volume of delivery platforms that the story can get released in the fastest time. No longer does a news and information organization depend solely on its over-the-air or cable-channel distribution. News depends upon a multitude of delivery forms to get the message (and its advertising) to the user.

Delivery requires speed. Data generated from a single instance of a story must be conformed to many additional form factors—web delivery, social media, streaming services for phone and mobile devices, cloud storage (as applications) for other users or subscribers and, of course, the primary service from which it bears their name and logo of identification.

Each of the elements for this delivery will depend on a reliable and effective storage solution and its delivery methods. Any single failure can create a cascading effect that is unpredictable due to factors such as time, impact, audience attention and more.

How the organization picks its storage architecture depends highly upon how, when and where the content must be delivered—and who needs to get that content first. Such content delivery channels (paths) may be internal: that is, editing, production, post production, immediate on-air studio playout or next-time/repeat delivery playout. For external delivery that need may be for users who only see the content in a linear format; or for nonlinear services that repurpose and re-present that content; and many other varying avenues of consumption.

COMPETING PROCESSES

Copying, replicating, reformatting, transcoding, asset management delivery and other components are often utilized in both sequential and random processes. Such processes may require competing storage architectures along with a storage delivery network that can adapt to sudden changes, bursts in services and an unusually high level of continual stress that is seldom predictable.

Thus, picking the “lowest common denominator” for the storage solution isn’t the best, most efficient or most advantageous choice. Storage architects who configure these platforms need to understand the requirements and develop a solution (or set of solutions) that best services each of these needs in a flexible and adaptable environment.

As an overview, storage architecture styles may include “takeoffs” or examples of the following kinds or levels of storage systems.

TIGHTLY OR LOOSELY COUPLED

A loosely coupled system will not share memory amongst or between its nodes (Fig. 2 Green). In this system, data is likely distributed among many nodes, which may involve a large amount of inter-node communications during data writes. While simple to use and for distributed reads where data can reside in multiple places, such a system can be expensive when looking at the system cycles metric.

Transactional data (non-mediacentric data) can be impacted by things such as hidden write locations that are effectively low latency and brought on by the types of storage, such as NVRAM or SSDs. In this model, data may be kept in more than one location allowing multiple nodes to hold it and speed up access based upon paths that might be open, even if momentarily.

Conversely, in a tightly coupled architecture (Fig. 2 Blue), data is distributed between multiple nodes, often configured for parallel running and orchestrated by several high-availability controllers configured in a grid. Here, inter-nodal communications is necessary to keep the efficiency level high, latency low and processing running at peak capacity. Often such systems are engineered so that I/O paths are symmetric amongst all associated nodes.

In similar fashion to how enterprise class network switches function, storage system failures (drives, controllers, memory) are quickly identified, and alternative (prescribed) paths are implemented almost instantly. The effect goes unnoticed and operations continue without impact.

MULTI-TIERED AND CLUSTERED

In this model, applications (such as HTTP-based calls) will manage and make appropriate use of separate, almost layered tiers for specific delivery platforms. Web, application, database (asset management) and processing (transcoding or packaging) servers already have the storage access pathing embedded in their code-bases. Calls to central storage are routine, secure and redundant. Should any one tier be compromised, the other paths will be protected (i.e., are safe) with the aid of network security protocols, firewalls and even managed, physical switch segregation.

In the clustered environment, memory is not shared between nodes and data will “stay” behind a single compute node (Fig. 2 Red). I/O paths in a clustered architecture model may have varying layers. Some may employ an umbrella-like or federation model allowing the entire system to scale out as necessary. I/O is manipulated until the appropriate node reaches the data set needed for the particular task. Redirection code manages these operations with a potential drawback that induces potential latency while the right path awards the right connection for the selected data requirements.

DISTRIBUTED ARCHITECTURES

When a nontransactional data model is necessary, as in data in an editing, media asset management or a random processing chain (that is a nonlinear and nontranscoding operation)—a distributed storage architecture model across multiple nodes may be the choice. In this model, memory is not shared amongst nodes and the data is distributed across the nodes as in a fanout or multiparallel infrastructure. From a file system structure, such a distributed architecture may use objects and may operate as a non-POSIX (Portable Operating System Interface per IEEE Std 1003.1-1988) protocol.

Distributed architectures are less common but sometimes still employed by large enterprises, which enable petabytes of storage (Fig. 2 Green). Search engines and extremely complex asset management implementations with federated access on a global basis are candidates for the distributed architecture. In this situation, massively parallel processing models are implemented, resulting in speed and scalability—a perfect example of cloud-based resources when the onramp/offramp accessibility is unencumbered.

There are still more considerations for storage architectures that involve cost evaluations, compromises and scaling requirements—sometimes based on the phase of a particular project or the impacts of legacy and existing functioning components that still have a financial life expectation sitting on the books. Working with an expert in these areas is essential, especially at the enterprise level. Software applications, workflows and data structures all become key components in determining the full solution set. When the delivery of your assets depends upon speed, reliability and performance, one cannot afford to do things only half way.

Karl Paulsen is chief technology officer at Diversified and a frequent contributor to TV Technology in storage, IP and cloud technologies.

Users of cloud services access a huge volume of capabilities, processes and opportunities when they open an account and start moving data through the system. These users—whether as businesses, individuals or government—are actively engaging in the continually changing digital transformation. This installment examines updates and concepts in what is referred to as a “composable” infrastructure.

Agility is just one of the core reasons for utilizing the cloud and its infrastructure. Other objectives include services that are “friction free,” with control systems that maximize available resources and systems that provide a peak ROI over an infrastructure organized using automated provisioning and intelligent managed resources. Each of these signify basic requirements and rationale for choosing a cloud solution, however, some still wish to have similar functionality in their own managed private datacenter.

Infrastructures composed of compute and storage are built upon a foundational network with onramps and offramps between “ground-based” users and a cloud-provider that interleaves on a fabric that glues together those systems, which vacillate between compute and storage.

Such an environment is being referred to as a “composable infrastructure,” that is, a fabric of Ethernet-based hardware and innovative software layered in a distributed network that claims no limitations (Fig. 1).

Fully integrated datacenter solutions consist of controllers, switches and other hardware steeped in a managed set of software subsystems that respond to the needs and objectives of the consumers. Missing are the traditionally inherent constructs found mainly in on-premises implementations, which serve only the single sets of solutions for which they were conceived to provide.

Conventional architectures for datacenters incur specific limitations based on how the topology of the switches and configurations are architected. For example, while it is possible to build a non-blocking fabric, such as in spine-and-leaf, it usually will require dedicating half the bandwidth in the top of rack (TOR) switches to that fabric, which in many designs and applications, is expensive.

PROTOCOLS, BOUNDARIES AND DISTRIBUTION

System solutions are generally designed to maximize the usage of the available fabric connections (or links). Those designs then must further model for failure possibilities while mitigating the probability of network loops. Protocol boundaries (L2/L3 boundaries) are used as considerations whereby L2 (Layer 2) traffic is confined for communications to leaf nodes, while using L3 (Layer 3) addressing for inter-rack traffic.

Evenly distributing bandwidth across the fabric is a target objective of the L2/L3 boundary. When the datacenter bandwidth cannot be evenly distributed or cannot consume bandwidths uniformly, load balancing will be necessary. Distribution models will be random, driving the network design to some form of equal cost multipath technology (ECMP). Hot spots become the result of uneven traffic in the network, which is managed using techniques such as OSPF (open shortest path first).

LATENCY VARIATIONS AND WORKLOAD SEGMENTATION

Latency is a direct consequence to variations in bandwidth. Latency should be deterministic and will generally be based on a specific workload. In leaf-and-spine, however, these are (generally) not the case.

Multistage traffic transport can take on varying paths when considered end-to-end yielding to latency variances of great proportions and unpredictability. Boundaries, which are often determined by the networking topology’s wiring, can be harmful to workloads.

A strategy to combat these variations is to use workload segmentation—something that the cloud-solution provider has baked into its own architectures given the variability and the multitude of applications expected (and managed for) in the cloud architecture.

CONTROL PLANE/DATA PLANE

Besides the above descriptions of latency, cloud-based bandwidths should be distributed based on workload needs. Bandwidth will be dynamic. Much like the loading required when the compute demand is high, and I/O is low, bandwidth among the compute-serving bare metal devices will be allocated according to the needs of the system. Conceptually this is better managed in a cloud-solution environment than in a firmly structured on-premises datacenter, simply because of cost-to-value unpredictability.

Datacenter topologies will typically utilize a control plane and data plane as their foundations. Often the datacenter’s control plane is tightly controlled. Only a limited set of protocols may be available and are usually not designed to facilitate external, user-defined and dynamic demands—ones that change upon need and are more likely designed for specific operational threads and models.

SDN solutions, for “software-defined networking,” are used to address known limitations by decoupling the control plane from the network itself. Implementations of SDN made in early network architectures used fixed assumptions about how and where traffic would flow in that network. Cloud solutions, however, continually manipulate the flows and adjust (using principles of SDN) to mitigate the usually tightly controlled architectures, thus mitigating the fixed “single-lane/road-like” architectures in a hardwired datacenter environment.

COMPOSABLE FABRIC

A relatively new approach to resolving several of the issues described in these previous discussions looks at addressing data plane, control plane and integration plane (i.e., automation-centric) issues holistically and individually—and based upon dynamic workloads. In a composable architecture, the planes will independently evolve and leverage each other dynamically. Here, the data plane takes on the needs of physical connectivity across the network via its topology model. The data plane, through the application of distributed software, handles the functions of packet (data) forwarding—collapsing and routing the entire system to a single building block.

In large cloud environments, intelligent routing distributes the pathing among the various connectivity components using composable—that is, capable of being assembled, alterable and then disassembled—networking, that found in fabric management components distributed throughout the cloud.

High-performance, advanced fabrics can be found in closed systems to enable high-performance, cluster-based compute architectures. Ethernet-and IP-based networks previously suffered from these capabilities, but no longer. Such new approaches are evolving as cloud-based datacenters take on new foundations to serve the needs of an evolving set of clienteles, workloads and demands.

PERFORMANCE UPSURGES

End-user customers, for the most part, are unaware of these accelerating background systems—they just see performance increases once their data gets into the cloud. Cloud providers can charge for the faster, more powerful services.

Alternatively, building a new on-prem datacenter is further afforded through similar interconnection solutions that leverage logic-based systems for TOR and rack-to-rack fabric topologies. Logical fabrics can be selectively placed throughout the system, thus utilizing composable software-centric control plane solutions (that is, an automated, self-managing set of software subsystems).

New opportunities continue to grow outside the basic SDN controller’s ability to define workloads through server-based APIs or other associate parameters. By employing embedded protocols, equal-cost algorithms let workloads be autonomously managed without requiring the end user to manually manipulate the data services or integration plane. Results are transparent, no workload-awareness capabilities.

Keeping the nuts-and-bolts necessities away from the user is a plus. Intelligent systems can now make monumental improvements in capabilities, driving the fluidity and flexibility of the cloud even higher. 

Karl Paulsen is chief technology officer at Diversified and a frequent contributor to TV Technology in storage, IP and cloud technologies. Contact Karl at diversifiedus.com.

If you’re thinking about shifting storage to the cloud or if expanding the local storage area network (SAN) makes better sense, knowing the impacts of both options is an important study in capabilities and cost management. Even if you think you know what cloud storage might cost, perhaps because you’ve done it before, be sure you’re updated on all the recent facts, variables and combinations of services before making a move in either direction. Cloud services, like SAN storage, is an evolving and frequently changing environment. 

ALWAYS A CHALLENGE

Expanding storage localized at the facility or in a cooperative datacenter has always been a challenge—add in the cloud options and you have much to understand. Selecting the type of storage has a direct reflection on its cost—either way. The storage performance desired is directly related to the volume of work activities and processing speed (the I/O) that you will need for the selected storage architecture and workflow.

For example, if you need to render a large set of animation clips, you will want fast access cache-like storage that can handle the throughput from the render engines without delay or latency. However, the life of that content, once rendered on that storage type, is relatively short compared to the volume size and length of retention for the entire set of finished, rendered files. Workflows will mandate migration of in-process render farm storage to secondary, longer-term storage to keep flows consistently moving from render engine to holding storage.

Costs for local storage are fairly predictable. For the cloud it becomes harder to cost and difficult to plan for. While cloud is fast to create, the total costs can go through the roof if not well-analyzed and controlled from the start.

CLOUD STORAGE COST BREAKDOWN

Cloud storage is a bit like going to the smorgasbord; many items are à la carte. Fees include monthly access, retention time, storage volume and/or the use of inherent capabilities of the store itself. Fig.1 depicts storage services and on-ramps to cloud services (via software-defined WAN) whose services may include storage access or may stand as separate items. 

Use it once and quickly—a short term “put it there and take it back out”—and you’ll have one price. “Put it there and leave it there” for a lengthy period of time—another price. Need rapid access to something you placed into a long-term holding pen (an “archive”)? You can watch previously expected low-budget costs to take off like SpaceX launching multiple satellites one at a time.

Storage types and the services available are common cloud cost factors. Each services menu may have different parameters. Simple, elastic, deep (e.g., cold or “glacier”) vs. high-performance, regional or multiregional physical locations each have different cost factors. Choices become decisions that the user needs to make, each having a differential base on near- or long-term requirements, and, in turn, are reflected in each cost.

Costs vary based on where the storage is located; that is, where the physical data centers are geographically. Global replication and access to the storage, which gives lower latency and another degree of safety, is great if needed. If offices are scattered all over the planet with needs for shared file access, the added costs may not be as painful compared to the work lost in having to wait hours to days for file recovery.

Capacity, based on monthly consumption, definitely affects cloud storage budgets. Should your storage needs be layered (tiered), how much is placed into which bucket and how many copies of each bucket are needed for protection and/or accelerated access will impact costs accordingly. Local, hot or redundant storage will be priced at “so many pennies per gigabyte” for a set amount of storage (e.g., the first 100 TB) on a “per month” basis. Incremental increases won’t see much change, but 10x increases will see a suitable cost decrease measuring around a few points (100ths of cents) per 100 TB.

DATA DELETION—IT ISN’T FREE

Just because you paid to put storage in the cloud, and/or to keep it in the cloud for some period, doesn’t mean you can just “get rid of it” without incurring a fee. Surprise! Sometimes the cloud vendor’s “hook” is to let you use their buckets for a few cents per 100 TB per month. The shocker comes when you either want it back—now; or if you don’t need it any longer and you want to dump it. Expect a bill to dump the storage based on your agreement.

Data at rest fetches one price, but dead data for deletion brings another. If you’ve paid to put the data into deep, extended storage (usually at a much lower cost than rapidly available recoverable storage), the exit strategy will likely be different than if you paid a higher price for rapidly accessible storage, took the data out and never put anything back in its “formerly” empty space.

Contract terms are key to storage costs from time-zero to time-end. Variables are based on volume, accessibility, minimum retention policy and more.

POLICIES, PROVISIONING—AUDIT AND MOVEMENT

Thinly provisioned policies or confusingly complicated requirements should raise red flags. Before signing, verify the service level agreement (SLA); e.g., how it predefines maximum capacities for specific storage instances. There’s no standardization for cloud SLAs, but there can be large variances based upon multiple factors including short-term and long-term associations (Fig. 2). 

Audit your cloud service billing to be certain the policies are followed and that your own needs don’t over or under exceed those requirements. Use available monitoring tools included in the service offering.

Be observant of cloud data movement and minimize it wherever practical. Public cloud storage should not be your primary backup, unless of course all your activities are 100% cloud-based. If using a hybrid cloud/local storage solution, take advantage of data minimization features. Deduplication can save costs by eliminating duplicate, redundant files before the data moves from on-prem to the cloud.

COMPLIANCE AND SECURITY ASSURANCE

Cloud computing widens attack surfaces in multiple dimensions. Know which side is responsible for security controls and assurances related to operations, carriage, storage and protection of the data entrusted to those services. Cloud providers usually take responsibility for physical security, business continuity, network security and disaster recovery (DR). Other security controls are likely the responsibility of the consumer.

What can or cannot be moved to the cloud is dictated by security and regulatory requirements guided by a raft of state, federal and international regulations. GDRP, the SOX Act, HIPPA (in medical) and other rights and acts can be complicated. If you’re not sure or willing to take the steps to figure them out, your better choice might be to keep your data stored locally. Don’t second-guess the value of your assets; be informed.

FACTORS GALORE

Add into the cloud vs. SAN (local) storage stew things like backup, recovery, automation, support, vendor prescriptions and lock-in, plus manageability and reliability, and you have a set of concoctions that just might make you think again about where, how and into “what” you place your most precious data.

You should also evaluate multicloud storage agendas. In this scenario, it’s critical to your business operations to determine precisely what data will be managed and where; how it will be stored; and how that data is transported from ground-to-cloud, cloud-to-cloud and back to ground again. Egress or access points may actually be bundled in the contract or the services might be discounted in order to offset third-party costs for access.

Don’t take strategies for storage, whether on the ground or in the cloud, lightly. Organizations expect consistency, uninterrupted performance and unprecedented reliability at a manageable cost. In an owner-provided on-prem SAN storage solution or an all-in-the-cloud alternative, be sure you know the expectations. Hire an expert outside entity to guide you through the marshlands ahead of making that final choice.

Karl Paulsen is CTO for Diversified. He can be reached at kpaulsen@diversifiedus.com.

IT departments continue to look for ways to improve benchmarks for end-user devices throughout the enterprise, and in the media-domain that is no different. For media, especially editorial production, the need to create content that is funneled from various resources depends upon the workstation or server capabilities, but more importantly on the storage systems’ abilities to deliver the data faster and unencumbered.

A decade ago, the hot topics for maximizing storage capacity were centered on the thoughts and needs of “enterprise data storage.” There seemed to be little distinction between “storage for media” purposes and “storage for enterprise data,” despite the radical differences between media’s needs for high accessibility, large contiguous file sizes and uninterrupted delivery to editing workstations. Data was distinctively divided between “structured” as in transactional data and “unstructured-data” found in video and audio media (see Table 1). Approaches to managing these divisions varied depending upon the storage platform (NAS, DAS or SAN) and the volume of data to be managed on a usage level.

The stack of requirements for data storage management and efficiency included tiered storage, data migration tool sets and techniques such as data reduction and thin provisioning. Storage resource management was also emphasized during those times, ahead of the full acceptance and availability of solid state drives (SSDs) or NVMe (non-volatile memory express) devices (Fig. 1).

Initially, tiered storage was proclaimed as a first step for many organizations. Tiered storage was regulated by a process that took the least used data and moved it to a lesser accessible format such as linear tape. That model gradually evolved from on-prem physical tape libraries to deep archives in the cloud. Sometimes tape was retained for disaster recovery and legal reasons. Over time, and in some cases, the cloud would all but eliminate the on-prem tape medium.

PROACTIVE SERVICES

Storage resource management (SRM) uses a proactive approach that optimizes the speed and efficiency of the “available drive space” on a SAN (storage area network). For transactional data, the focus was on administrative procedures, which would automatically perform data backup, recovery and analysis. SAN solution sets often required a higher level of administrative maintenance, so SRM was furthered by implementing a combination of vendor APIs and a collection of the usual tools for systems management including SNMP, storage management initiative specifications (SMIS) and RESTful web services.

Utilization pattern data also emerged during this 2010 period. The tool set helped administrators determine how their storage systems managed input/output (I/O) requests. Through these tools, the systems could be improved by adjusting the way the drives were being utilized per the workflow groups to which they were assigned. Plug-ins were used to track real-time and trending patterns over any level of granularity (days, weeks or months) that the administrator wanted. For example, in a high-level rendering process for visual special effects, the demands on drive I/O would be continual but in editorial or color grading, the demands may be somewhat reduced.

LESSONS LEARNED

Many changes occurred in storage media between 2000–2010. The industry had started shifting SSD applications to include NVMe (2009) as the front end or cache portions of the drives arrays themselves, learning new lessons in space, speed and product fabrication. Even the migration from Tier 1 high-performance Fibre Channel (FC) deployment to less expensive Tier 2 SAS drives became more acceptable because improvements in the storage management tool sets allowed a more automated approach to implementation and administration. And once SSD arrived, the division between Tier 1 and Tier 2 almost blurred because I/O performance for SSD nearly equalized FC HDD performance, which the applications developers jumped on as a new and improved way of increasing performance of their own products.

Some of the lessons learned really leveraged rapid changes in drive performance alongside the acceleration of prolific content generation for streaming services—and for good reasons.

FAST FORWARD

A lot of things have changed over the course of the previous decade not only in storage device technologies, but also in performance, operations and administrative freedom as represented by shortened deployment time and user-sensitive interaction with storage management itself. Fig. 2 shows the relative transfer rate improvements from HDD through today’s NVMe storage sets.

Workflows are individual to the organization. Drive system providers are now focusing more on performance by adding simplicity to installation, configuration and administration. Features such as role-based authentication and single namespace architectures are taking the place of complex administrative activities that traditionally were the Achilles’ heel of the end users and which often kept systems administrators from going home at night or enjoying their weekends.

Software (not hardware) RAID is now supported by separating the controller profiles from the shares or pools of storage itself. Eliminating the dependency among the two systems enables yet another level of performance improvement. Desynchronization and data corruption is further prevented when using software solutions for RAID control itself.

Improvements such as hyperscalability, high availability (HA) and containerization support through application-specific interfaces are now easier and achievable, aided in part by taking the mystery out of the deployment factor. Through the addition of increased RAM and multiple CPU cores, more throughput and reduced latency are each achieved. Distributed file systems and clustering for media-centric storage implementation, typically reserved to high-profile compute intensive systems, is now expected.

Utilization of current optical fiber SFP connectivity—such as QSFP28 100G connections between the server engine node and the drive chassis itself—are pushing I/O (and IOPS) figures upward while being more readily adaptable to the user software systems for applications such as rendering, color grading and editing.

REMOVING RELUCTANCY

Nobody wants to or can afford to wait and with storage bottlenecks virtually eliminated using these newer and faster speedways—throughout the system—they are gaining acceptance across the enterprise.

Traditional IT departments who were reluctant to put different or new solutions in place, especially for M&E users, are changing their vision. Today, for example, deployments of new systems take only hours, not days. GUIs are easier to understand. Systems and their administration are more intuitive and quite different from what were previously used in older, established storage solutions.

Today’s users should expect a storage system to be as straightforward as their iPhone or Android mobile devices. Resiliency should not have to depend upon continual monitoring and tweaking of systems just to keep workflows fluid.

KNOWING THE RIGHT SOLUTION 

A key to understanding cost-to-performance models is in knowing the architecture of the storage system that you may need for your particular application. Stakeholders in the organization need to homogenize the users, administrators and technical support personnel in order to reach the right solution. A single drive set may no longer meet all the needs of the organization, however, that decision really depends upon the details of scale, diversity and the performance of the drive set and the solution provider’s unique capabilities for the tasks at hand.

Note: Portions of the technology discussions are courtesy of OpenDrives LLC of Culver City, Calif. 

Karl Paulsen is currently the chief technology officer for Diversified and a frequent contributor to TV Technology in storage, IP and cloud technologies. You can reach Karl at diversifiedus.com. 

If you are a media creation entity, you want to leverage as many potential opportunities as possible to progress through the stages of content creation. One of those is the development of a repeatable set of requirements, focused on specific workflow needs, such that operating in a “routine” mode is more easily achieved.

The ability to customize or replicate those functioning modes is advantageous when running multiple sets of processes simultaneously or independently. Cloud services or on-premise datacenters can provide effective conduits for such opportunities; however, having to reconfigure based upon systemic changes in the infrastructure can be time consuming, complex and require specialized resources especially for routine processes and simple updates.

Provisioning and managing datacenters through machine-readable definition files is the premise of what is known as infrastructure as code (IaC). Rather than supporting direct physical hardware configurations or solutions built on interactive configuration tools, IaC uses computecentric, machine language-based “files” to manage those compute processes.

In a cloud-based solution set, IaC deploys resources using templates, i.e., files that are both human-readable and machine-consumable and that instruct the systems to autonomously configure their functionality virtually automatically. Cloud service providers offer such IaC solution sets as a “built-in choice”—one that a user may use or ignore.

Fundamentally, once a code template is created, the cloud system then takes those code instructions and administers them to the cloud’s resources without any further direct user intervention. Any needs for the updating of called-out resources or for replacing any of the processor chains to achieve goals is handled as a background function and essentially become a “hands off” operation. Fig. 1 depicts the workflow basics from the user through the services, whether in the cloud or in an on-prem datacenter.

BENEFITS TO IAC

Benefits to the applications and uses of IaC include visibility, stability and scalability. Others include security, verification, repeatability and extensibility.

Repeatability, with security, is achieved when the same settings are utilized in each instance of the template. Verification that a given provisioning is stable and ready to run assures that if there is a failure, the infrastructure can be rolled back to a known state without a catastrophic collapse of the components. Operations can continue or be temporarily suspended depending upon the prescribed workflows.

Visibility lets the user obtain a clear reference point to what resources are being used on the account. Should something inadvertently change—such as a wrong setting or an accidentally deleted resource—the stability mechanism utilized in an IaC deployment can help resolve that change using a combination of a current or a previous control management version.

Scalability is equally important. Building a library around reusable code sets lends to the templated model, which can be easily and readily distributed to multiple services globally. Should a particular region need to ramp up for an unexpected deliverable, the closest cloud port could rapidly spin up the services and the infrastructure, based on the templates likely in use at another geographically distanced site. Users would not necessarily need to transport data to an alternate site if the repository can be brought into service in another region.

Fig. 2 diagrams where templates, scripts and policies are held in a common repository, which can be appropriately relegated to each global point-of-presence, i.e., a cloud zone or datacenter. Each of the practices can then be pushed into (or pulled from a repository) to the associated locations and functions.

EVERYTHING AS CODE

A similar approach is the practice of treating all the components of the solution as code. By storing configurations along with source code, in a repository and as a virtual environment, code sets can be cycled or recreated whenever needed. Even system designs would be stored as code in this model.

The everything as code (EaC) model mitigates the need for physical hardware and connections to be installed for each functional activity or task. This obviously would be impractical—and impossible—in a cloud-centric atmosphere. Thus, the previously required specialized physical skill sets and designer practices are transformed into a code-ready environment.

Native cloud applications once relegated to physical modifications have changed the entire cost model, making it easy to spin up a “virtual” infrastructure foundation regardless of location.

FAMILIAR STATEMENTS

Like IaC, an EaC model has similar beneficial statements. Repeatability, including the ability to move from one cloud provider to another, allows for the precise recreation of the environment that can further leverage new feature sets (such as faster performance or less cost per cycle). Tested infrastructure code can be developed, validated at scale (through compute modeling), and then directly promoted into production with the expectations, confidence and assurance it will function quickly and as designed.

The fear, uncertainty and doubt factor (FUD) with respect to server configuration drift is all but eliminated. These new models can literally self-heal themselves to almost any level—including a complete redeployment should a server die or need patching for continued operability. Since the entire infrastructure is developed in code, a mirror image of the system with no crossover dependencies can be spun up the moment an anomaly is detected. Operations just keep running.

INFRASTRUCTURE TOOLS

For cloud solutions to be practical, they need to be dynamic. Infrastructure resources fall into that category. It is akin to having infinite patching and shuffling capabilities without having a human actively manipulating the functionality. Each cloud provider is likely to have their own “flavor” of either IaC or EaC depending upon their feature sets.

Such tool sets allow cloud customers to specify their needed infrastructure resources without having to actually understand (logically or physically) how they are interfaced to one or another. The tools further let the users allocate which resources are needed, the parameter limits (how much for how long), and how those resources should be configured to perform selected tasks and activities.

In platform as a service (PaaS) architectures, users could use a particular platform’s user interface to assign or create resource sets and then manage those resources throughout its operations. In similar fashion, third-party solutions providers would make graphical user interface (GUI) products to manage both cloud and virtual infrastructures and sell those products to consumers. The drawback, however, was these were essentially “constrained” (specific) services that required substantial investment in initial specifications, design and testing before they could be rolled out into service.

While arguably the PaaS practice is practical once configured—and could be likely transported to various other cloud providers—the model was not as flexible. Apps required maintenance and upkeep when a systemic change in the cloud’s internal models were updated. Sometimes the changes impacted the PaaS applications and sometimes the PaaS would “self-adapt.” It was all about the type, use and applications, which were deployed at that time for that particular service.

CODE EXPERTISE EVOLVES

With open access to the virtual “moving parts” of the cloud, IaaS and PaaS models are changing. Where once codebased development was limited to a set of code-level experts, the new era is evolving to integrate machine learning and human-readable practices to become more prevalent and more productive.

Early adopters of cloud services recognized the needs for dynamic infrastructure platforms and are now changing their internal applications to implement their own self-provisioning and configuration capabilities. For those systems housed in private (non-public-cloud) datacenters, once the user/operators learn about processes, patterns, practices and accessibility, they can eventually orchestrate their own server structures, build their own server templates and promote the ability to update running servers without disrupting operations.

Karl Paulsen is the chief technology officer at Diversified, a SMPTE Fellow, and a regular contributor to TV Technology. You may reach Karl at kpaulsen@diversifiedus.com. 

Security guidance for ecosystems associated with data and storage systems had for many years focused principally on the protection of their associated systems, such as hardware, connections and processes used in backup or duplication. Other guiding objectives stayed fixed on the general support of evolving information security standards, per ISO/IEC 27000 directives and a collection of closely bound and integrated documents.

Standards have played important and strong roles in molding information technologies to become a harmonized set of criteria that has guided hardware, software and implementation. As a part of the entire ISO/IEC 27000 family of standards, a specific suite of documents is published by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) under the joint technical committees (JTC) in ISO/IEC JTC 1/SC 27 for IT Security techniques. The ISO/IEC established a series of continually developing documents, which grew out of industry’s rapid growth in IT, storage and networking.

The JTC was founded in 1990 when an earlier subcommittee (JTC 1/SC 20) moved outside of the field of security techniques associated with phrases such as: “secret-key techniques,” “public-key techniques” and “data encryption protocols.” These late 1990s techniques were later disbanded in favor of emerging techniques driven by new products, advanced needs and obsolete applications.

As is often the process in standards development, these root-1990 subcommittees, and beyond into modern day, have systematically altered various groups’ scopes and associated working groups’ efforts to meet current (at that particular time) demands for standardization. As such, numerous details, trials and tribulations often appear in more up-to-date applications as the technologies mature, emerge or obsolesce.

Data, in the form of assets, is of paramount importance to individuals, industry and enterprise. Many understand some of the founding principles (backup, for example); yet others have no foundational practice or reasoning as to why one approach is taken compared to another. This issue opens the topic of storage security as applied to using some of the ISO/IEC standards to support the safe keeping of media-related data and its practices.

CORE DATA PROTECTION

Like most network or storage system administrators, there are three core issues associated with data protection: privacy, information security and assurance, plus the storage (data) itself (Fig. 1). The goals are associated with protection centered on the notion that “all security goals and objectives be continuously maintained irrespective of any system complexity, variation in performance or the granularity of individual fidelity”—that is, without compromise.

Assuring all these goals are maintained is paramount to elevating the sustainability of the system, the integrity of the storage (i.e., its data) and the appropriate level of privacy, protection and information at all levels. When one realizes the broad areas such approaches might need to address, the details could be daunting. And this is precisely why the ISO/IEC groups assembled a series of documented guidelines and processes by which to follow.

However, the approaches can be confusing depending upon which versions or at what time your planning was first introduced. For example, in storage security guideline ISO/IEC 27040, the concept of “data protection” is not specifically addressed, despite a high degree of insight, which is presented in the document. Instead, this standard aims to establish security controls, which in turn helps elevate awareness of storage security through additional feature sets and best practices.

DATA PROTECTION CONTROLS

Since the starting point, file-based data workflows for media, audio/video and motion picture production depended upon and utilized simple storage management solutions with a modest degree of structure for its direct data protection. Early in that progression, users leveraged external storage solutions (e.g., removable drives) as a fundamental backup solution. Network storage was a “few and far between” alternative. These methods were sufficient, initially, but changed dramatically as storage volumes accelerated exponentially.

Likewise, only a modest number of users (and systems) had any serious concern for data protection—primarily because self-management and isolation was an easy, relatively straightforward process unencumbered by hackers, pirates and disruptors at that time. Hardware integrity posed more problems than content stealing prevention, due in part to proprietary codecs and self-regulated, insulated code support channels

Today, of course, we have an entirely different perspective. Exponential volumes of data have driven business continuity and disaster recovery requirements upward. Backup copies are now built to protect against data loss, one of a continually expanding set of data protection controls. Implementation guidance outlined in the ISO/IEC 27000 series (Table 1), prescribes organization requirements for information backup, including, but not limited to, software, systems and management policies.

Essential information beyond simply the data must be retained in facilities that can be run with the appropriate hardware and software tool sets to recover duplicate data sets should the original hardware be compromised. Recovery processes should be documented, itemized and their implementation procedures routinely tested should the main facilities be lost or destroyed.

Operational processes, such as how to monitor backup execution, must be reviewed against new software application versions whether locally hosted or in the cloud.

SECURING BACKUPS

At first the concept for securing backups, documented in 2013 under ISO/IEC 27002, may seem a bit archaic in the 2020 era of cloud-based protection, but they should not be forgotten or discarded by any means. In fact, the requirements for storage security, related to (secure) backups, are now clearly ingrained in the ISO/IEC documents.

Backup security is just one of the pillars for overall security on a broad scale. Today, it makes further sense to have a functioning system “in the cloud,” yet it does no good if the procedures are not consistent. Cloud vs. datacenter practices should often both be reviewed; have their routines updated and harmonized; and assure that all the code “tricks” created by developers remain in line with each other.

Extensions to storage security practices, which may now be ported to a cloud environment, could also be replicated in another “offsite” datacenter. Cloud may be for deep archive, with the data center being for rapid recovery—the choice is up to the business owner and administration.

TRUST AND VERIFY

Updated extensions include guidelines from a 2016 update of ISO/IEC 27004, which outlines “the effectiveness of measuring information security.” The 2016 update completely restructured the previous (original) document expanding it with a new purpose and putting it into current rules per the ISO/IEC Directives-Part 2.

Specific changes identified in the updates now lean toward media encryption, and operator authentication and authorization. They harmonize security practices with specifics identified for both backup systems and storage media. The concept of the “trusted” individual, component and/or system is now added, with the inclusion of a cleared and vetted or bonded individual.

Audit trail identification for backup processes and a clear path to verify the backup was actually performed are now required. Furthermore, a means to physically prove that restoration requirements are being met is necessary for proper certification. In effect, the backup is now trusted and the processes are now verified, per the standards.

FINER POINTS

ISO/IEC standards go much deeper and can be studied if you have access to or subscribe to the specific documents—most of which are not free. In part, data availability recommendations will include procedures for reliability, fault-tolerance and the requirements for performance of the data itself. Users must insure (and assure) against unauthorized access using technologies such as “data in motion” encryption, i.e., data that is in transit or in flight, is encrypted when the data is in the process of being moved (transported) between locations in either the facility, the computer and/or the network (Fig. 2). 

Like the practices and procedures found in well-known industrial efforts, such as in ISO 9000-certified facilities, should have similar concepts and approaches to those well entrenched in IT and storage solutions. Each of the ISO/IEC-subcommittee’s agendas helps to support the practices and provide sustainability across the industry with consistency. Appropriate follow through by all the storage disciplines should be expected. 

BLOCKCHAIN CHANGES THE COMPLEXION

Finally, we hear a lot of new uses and applications of blockchain, which is making data more secure, again. Transactional interchange of data that is untraceable, inaccessible and only known by the sender and the designated receiver—irrespective of the number of parties—is changing how content, contracts and transactions are being handled. The interchange of that data is so private and so secure that it is highly likely it will change the rules of storage and security forever. Stay tuned for upcoming topics on blockchain and how it will make the world a safer place for technologies.

Karl Paulsen is a SMPTE Fellow and frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at ivideoserver@gmail.com. 

What is the cloud, really? “Special Publication 800-145 (September 2011)” from the National Institute for Standards and Technology says to be considered a “cloud” there are essentially five characteristics that must be included. Cloudspotter’s Journal will explore these five characteristics according to the descriptions provided in the NIST document and including other applicable definitions, applications and terminologies from the consumer/user perspectives.

ON-DEMAND SERVICES

For users to validate a cloud provider’s services there must be the ability to unilaterally provision certain computing capabilities, e.g., server cycles and time, applications or network storage. Services must be deployed in an automatic fashion and without human interaction. The automatic, on-demand services are run under certain scripting commands, using abstraction principles administered under “orchestration” components that are aware of the system resources, locations, as well as the demands by other users or services throughout the cloud environment.

Cloud services may be either thick or thin client platforms. The thick, sometimes called “fat” clients, are those clients that will perform the bulk of the processing—as described in general client/server applications. The “thin” client often refers to the software, which describes the networked computer itself. The thin client is software designed to enable communications amongst the servers.

NIST described the “broad network access” as capabilities available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms. Cloud clients include computers, e.g., workstations, tablets, laptops, smartphones and other mobile devices. Users access the cloud services by connecting to these networked cloud client devices.

The third qualifier enables the doling out of compute resources in a pooled fashion that can serve multiple consumers using a “multitenant” model. Each tenant’s data is insulated from any other tenant’s data—essentially remaining invisible to other operations or tenants.

NIST further explains that the model pools different physical and virtual resources, which are dynamically assigned and reassigned according to consumer demands. Customers are completely unaware of the location of the resources and have no control or knowledge of the exact locations of the provided (or available) resources. Resources, in these cases, include storage, processing, memory and network bandwidth.

RAPID ELASTICITY

Elasticity is the degree to which a system can change and adapt. In cloud applications, this applies to workloads and includes the ability to move from one use configuration to another and then back, or to yet another, depending upon demand. The term “rapid” is added in the NIST profile to include the system’s capability to be provisioned and released, to scale rapidly outward and inward commensurate with demand.

To the consumer, this provisioning appears to be unlimited, allowing services to be appropriated in any quantity at any time. The degree of elasticity then becomes transparent and is governed by the consumer’s (user’s) expectations for a deliverable service (infrastructure or platform) based upon cost, quantity and quality.

MEASURED SERVICE

The last of the five essential characteristics is that cloud systems will automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service. Again, these services could be storage, processing, bandwidth and the activity of the specific user accounts.

Resource usage can be monitored, controlled, and reported to the user and the provider. This continues the transparency (of the services) for both the provider and the consumer of the utilized service.

In computer science and software engineering, the principles of abstraction are designed to reduce complexity and ensure efficiency in complex software—as in cloud-based—systems. A common theme of cloud services is that of “abstraction through the processes of virtualization.” Abstraction is described as “existing in thought or as an idea but not having a physical or concrete existence.”

In a cloud-based environment there is a “physical” existence for the compute (GPU, CPU, servers), the network, storage and all the associated software components. All these components are distributed so that in the event any one location should fail or be short of resources for any reason (i.e., for maintenance, failure or updates), the general topology of the entire cloud “network” will pick up those services and continue transparently and without interruption.

In creating shared pools of resources, there is an abstraction mechanism that maps a logical address to a physical resource. Cloud computing networks utilize various techniques, which they develop to create virtual servers, virtual storage and virtual networks. Depending on your definition, virtual applications are also available to any device and provided by the cloud providers.

Abstraction enables the key cloud computing benefits of shared, ubiquitous access irrespective of location or demand by others.

CLOUD SERVICE AND DEPLOYMENT MODELS

NIST describes the service models and the deployment models for cloud computing (Fig. 1). Principally, the three most familiar service models are Software as a Service (SaaS); Platform as a Service (PaaS); and Infrastructure as a Service (IaaS).

SaaS allows the consumer to use the cloud provider’s applications and services, which run in a cloud infrastructure, but prohibits consumers from managing or controlling the underlying infrastructure or its individual applications, with the possible limited exception of users-pecific confituration settings.

PaaS allows the consumer to deploy consumer-created or acquired applications, which are created using the programming resources (tools, languages, libraries and services) and are supported by the cloud provider. In the PaaS environment, consumers do not manage or control the cloud infrastructure (i.e., the network, servers, operating systems, or storage), but the consumer does have control over the deployed applications possibly including the configuration settings for the application-hosting environment.

Consumers, in IaaS, may provision processing, storage, networks and other “fundamental” computing resources where the consumer is able to deploy and run arbitrary software. Again, the consumer does not control or manage the provider’s underlying cloud infrastructure, but they may have control over the operating systems, storage and the deployed applications. Select networking components, such as host firewalls may be controlled on a limited basis by the consumer.

Cloud computing is an evolving paradigm with an unknown actual starting point and an open door into how compute-centric business will prevail.

Irrespective of where or when the term or the practice was created, there remains no doubt that “cloud” is here now and will remain the future for an untold number of users and applications.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at kpaulsen@diversifiedus.com.

Deciding, for the long run, where to keep an organization’s media-centric assets (i.e., original content, EDLs, finished masters, copies, versions, releases, etc.,) is reshaping the ways of storing or archiving data. Continued popularity in cloud-based solutions for ingest/content collection, playout from the cloud and processing in a virtual environment leads many to rethink “storage in the cloud.”

Yet cloud concerns still leave the door open to storage alternatives, tier-based migration, automated management and more.

What are the possible alternatives: Backup or archive? On-prem or in a co-lo? Private, hybrid or public cloud? Knowing the differences could create changes in how to approach archive management, regardless of the size, location or types of data libraries.

BACK IT UP OR ARCHIVE IT

Let’s look first at the differences in backup vs. archive.

Backup is a duplicate copy of data used to restore lost or corrupted data in the event of unexpected damage or catastrophic loss. By definition, all original data is retained even after a backup is created; original data is seldom deleted. Most backup just in case something happens to the original version, while for others, it is a routine process mandated either by policy or because they’ve previously suffered through a data disaster and pledge never to live through that again.

On a small scale, for a local workstation or laptop, practice suggests a nightly copy of the computer’s data be created to another storage medium, e.g., a NAS or portable 4-8 TB drive. Travel makes this difficult, so alternative online solutions prevail.

Businesses routinely backup their file servers (as unstructured data) and their databases (as structured data) as a precaution against a short-term issue with data on a local drive being corrupted. “Snap-shots” or “images” of an entire drive (OS, applications and data) are often suggested by administrators, software vendors, and portable hard drive manufacturers.

Incremental backups, whereby only new data or any which was changed since the last backup, are made due to the larger storage volumes and the time required for the full backup.

STORAGE AS A SERVICE

Archived data could be placed on a local NAS, transportable disk drive, an on-prem protected storage array partition, or linear data tape. Since an archive is about “putting the data on a shelf” (so to speak), the choices vary based on need.

Cloud archiving is about “storage as a service,” and is intended for long-term retention and preservation of data assets. Archives are where the data isn’t easily accessed and remains for a long uninterrupted time.

Archiving used to mean pushing data to a digital linear tape (DLT) drive and shipping those tapes to an “iron mountain-like” storage vault. In this model, recovering any needed data was a lengthy process involving retrieving the information from a vault, copying it to another tape, putting it onto a truck and returning the tape to the mothership where it was then copied back onto local storage, indexed against a database (MAM), and then made available on a NAS or SAN.

This method involves risks, including the loss or damage to tapes in transport, tapes which went bad or possibly had corrupted data onto the tape in the first place. Obsolescence of either the tape media or the actual drives meant that every few years a refresh of the data tapes was required, adding more risk in data corruption or other unknowns.

As technology moved onward, robotic tape libraries pushed the processes to creating two (tape) copies—one for the on-prem applications and one to place safely in a vault under a mountain somewhere. While this reduced some risks—such as placing duplicate copies in storage at diverse locations—it didn’t eliminate the “refresh cycle” and meant additional handling (shipping back tapes in the vault for updating). Refresh always added costs: tape library management, refresh cycle transport costs, tape updates including drives and physical media, plus the labor to perform those migrations and updates.

STORAGE OVERLOAD

Images keep getting larger. Formats above HD are now common, quality is improving, pushing “native” format editing (true 4K and 5K) upwards, added to dozens to hundreds more releases per program and keeping storage archive equations in continual check. As files get larger, the amount of physical media needed to store a master, such as protect copies and one or more archive copies of every file, is causing “storage overload.” Decisions on what to archive are balanced against costs and the unpredictable reality that the content may never need to be accessed again.

High-capacity, on-prem storage vaults can only grow so large—a hardware refresh on thousands of hard drives every couple years can be overwhelming from a cost and labor perspective. Object-based storage is solving some of those space or protection issues but having all your organization’s prime asset “eggs” in one basket is risky and not very smart business. So opens the door to cloud-based archiving.

MANY SHAPES AND SIZES

Fee-based products such as iCloud, Carbonite, Dropbox and such are good for some. These products, while cloud based, have varying schemes and work well for many users or businesses. Private iPhone users get Apple iCloud almost cost-free but with limited storage sizes. Other users prefer interfaces specifically for a drive or computer device with an unlimited storage or file count.

So why to pick one service over another? Is one a better long term solution versus another? Do we really want an archive or a readily accessible “copy” of the data in case of an HDD crash?

One common denominator to most commercial “backup/archive” services is they keep your data in “the cloud.” Data is generally accessible from any location with an internet connection and is replicated in at least three locations. However, getting your data back (from a less costly archive) has a number of cost-and non-cost-based perspectives. Recovering a few files or photos is relatively straightforward but getting “gigabytes” of files back is another question. So beware of what you sign up for and know what you’re paying for and why.

Beyond these common points is where the divisions of capabilities, accessibility, cost, serviceability, and reliability become key production indicators which in turn drive differing uses or applications.

ACCESS FROM THE CLOUD

Cloud-stored data costs have a direct relationship to the accessibility and retrieving of that data.

If you rarely need the data and can afford to wait a dozen hours or more for the recovery—then choose a “deep-storage” or “cold storage” solution. If you simply need more physical storage and intend regular daily or weekly access—then you select a “near term” (probably not an archive) storage solution. Options include On-Prem (limited storage, rapid accessibility) or off-prem in a Co-Located (Co-Lo) storage environment, referred to as a “private-cloud” or alternatively a “public” (or commercial) cloud provider such as Google, Azure, AWS, IBM or others.

Cloud services continue to grow in usage and popularity, yet, there remains a degree of confusion as to “which kind of cloud service” to deploy and for which “kinds of assets.” Many users prefer regular access to their “archived” material—this would be a wrong approach and is more costly (as much as 4:1) than putting their data into deep-or cold-storage vs. a short-term environment (“a temporary storage bucket”) that is easily accessible.

Fig. 1 shows a “hybrid” managed solution with local cache, multiple cloud storage providers, and local/on-prem primary archive serviced by an object-based storage “bucket” manager. The concept allows migration, protection, and even retention of existing storage subsystems.

CHOICES AND DECISIONS

Picking cloud service providers for your archive is no easy decision—comparisons in services and costs can be like selecting a gas or electricity provider. Plans change, sometimes often. Signing onto a deep archive becomes a long-term commitment due primarily to the cost of retrieving the data despite the initial ‘upload’ costs being much lower. If your workflows demand continual data migration, don’t pick a “deep” or “cold-storage” plan; look at another near-line solution. Be wary of long-term multi-year contracts—cloud vendors are very competitive, offering advantages that can adjust annually.

The amount of data you’re to store will continually grow. Be selective about the types of data stored, the duration you expect to keep that data, and choose wisely as to ‘what’ is really needed. Your organization’s policies may dictate “store everything,” so know what the legal implications are, if any.

Carefully look at the total cost of ownership (TCO) in the platform, then weigh the long-term vs. short-term model appropriately.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

The phrase “the future is now” can be no more reflective than when you consider the merging of the cloud with IP and virtualization. This Cloudspotter’s Journal looks at changes that are reachable and already on the horizon. The merging of IP (in various flavors) with virtualization, and their machines, are creating a new environment that yields suitability for cloud production, continuity playout or consolidation of operational resources.

With the continuing maturity of the latest IP standards for high bit rate—professional media on a managed network—comes the question “Could this be applicable to ‘the cloud’?” If so, just how would you sandwich 1.5 to 12 gigabits per second onto a public highway in an efficient and productive way? Today, this is likely an unsubstantiated perception; yet, there is certainly an assumption given that everything else is headed in that direction, so why not IP?

Such an assumption might be valid if you had an unlimited budget and you were physically parked next to Azure in Redmond with direct-to-cloud fiber connectivity that didn’t need to go any further. Even with that “pipe dream” the security issues and egress management alone would likely put a halt to that nonsense in a heartbeat. Pipe dreams aside, alternatives for high bit rate data movement and manipulation still need to be realized.

CLOUD ORIGINS

What if the cloud went back to what it originally started as? The actual cloud symbol—that puffy squashed circle-like icon—was just a representation of “the network” on a diagram (Fig. 1). The network could, at the time, be described as anything from a short-connected Ethernet segment with files moving on it to a full datacenter or anything in between. It might have been a campus-like topology infrastructure where resources were easily shared; that is, doled out, when or as needed and then reconfigured to serve other purposes once that “compute cycle” effort was completed.

Then consider how a “shared resource” model, as in a datacenter, would be analogous to the cloud. A shared resource model, when appropriately configured, is a cloud; with on-premise cloud versions functioning like a datacenter. Pools of resources are allocated as needed with a central management and monitoring platform overseeing operational functionality (sometimes called “orchestration”). How the services are efficiently and effectively abstracted from the resource pool becomes the challenge.

The “co-lo” (co-located) facility is similar, but they are not generally “on-prem located.” In this model, a datacenter-like structure houses several screened-off areas (i.e., private “cages”) where customers place their own equipment and connect it to the internet or a similar network topology (Fig. 2). Sometimes the co-lo equipment cages are managed independently and sometimes not. The co-lo is usually connected at a main internet point, whose point-of-presence (POP) is usually very close or actually co-located in the facility.

Nonetheless, the co-lo concept appears a bit like a cloud, without the customary sharing of resources found in a dedicated pooled resource environment.

Co-lo datacenters usually service many customers. However, they only become a true “cloud” if the individual sets of components (from the individual customers in their own cages) can leverage one another, pooling their collective resources into a managed, coherent system of compute resources. This doesn’t happen very often, unless it is a campus research center where the cages are interconnected to/from a central processing core that can allocate and then reallocate their functionality to serve varying purposes.

Co-lo models have been applied to broadcast operations, such as in Jacksonville, Fla., where a datacenter approach first served Florida Public Broadcasters and then later added customers from outside the Florida region. This was known in its early days as “centralcasting” and was adapted to yield the “hub-and-spoke” concept for consolidation of television station playout and content ingest. The model was extended to differing formats, but it is essentially a cloud-like central facility, which sometimes shares its resources in a virtual environment and sometimes houses discrete components assigned to specific distant station markets but managed (i.e., controlled) from a single location.

SHARING THE RESOURCES

Shared-resource private-cloud functionality, as found in current and emerging environments, seems better served when the physical architecture of the supporting components (switches, servers and compute topologies) is virtualized to support multiple functions depending upon the day-part needs for that playout region, city or operations center. Such capabilities, promoted by broadcast equipment manufacturers as products that were once built into application-specific and dedicated functionality (e.g., an up/down/cross converter, graphics generating device, render engine) are moved away from purpose-built hardware to software-based products landed on universal “pizza-box” servers.

As FPGAs and other compute devices (such as GPUs and multi-core CPUs) evolve technically, manufacturers are likely to move further away from single purpose black boxes to a shared services model on COTS equipment. Manufacturers already recognize the capability to port functionality into a series of blade processor servers, or such. This model is becoming more viable to end user’s applications (Fig. #3), where a server ‘pool’ is repurposed depending upon facility needs).

For traditional facilities, where the hardware gets only about a 25-35 percent average utilization factor, the concept is to move from a ‘dedicated’ (per device) model to a ‘shared’ (i.e., pooled) environment. The model is now considered virtualized, with the devices (the bare metal COTS servers) becoming multifunctional through a software defined architecture. In the ‘cloud-like’ model, the functionality of a pool of shared compute engines, for example, can be spun up to transcode for a portion of the day and then spundown, reconfigured (in software) and spun-back-up to do another function perhaps as a format converter or MAM processing engine at another time. As more content is generated in a single cloud, the model then expands to provide services to multiple channels (distribution points) and thus, the cost-per-channel goes down considerably.

FULL FUNCTION FLEXIBILITY

Media organizations are now starting to head in the direction of cloud-flexibility and in mixed locations such as on-prem, hybrid and private or public cloud. While in the past, functions were optimized for compressed workflows (e.g., DNxHD, XDCam, AVI formats), the capabilities of high-speed NICs and faster processors have enabled workflows in full resolution, uncompressed formats. Some systems now coming online are utilizing the recent SMPTE ST 2110 IP studio production standards for formats from HD (1.5 Gbps) to UHD (4K) at upwards of 12 Gbps.

Returning to an earlier discussion, a cloud utilization model now makes practical sense whereby owners can create an on-premises, cloud-like architecture that supports full bandwidth (HD/3G or UHD) and can serve many entities locally and afar. Here processing is done in full bandwidth and then distributed (as completed work) in a compressed format suitable for air or CDN/distribution. This spares each local facility from building up services that take up expensive real estate, yet only get used a fraction of the time.

Extending these capabilities to full-spectrum, high-quality production capabilities that support end-to-end services is gaining momentum on a global scale. Remote (REMI or “at home”) productions are now carrying out field production needs that once required a fully equipped mobile (outside) broadcast unit and a crew of dozens to produce sports and entertainment. Outfitting the datacenter to share large sets of servers in a network topology can now raise the utilization factor manyfold, while reducing the overall capital expenditures (and labor costs) that were once required at every station or venue location.

Virtualized environments, whether on-prem or in the cloud, make a lot of sense as facilities consolidate locations and share their resources. Deploying in a virtualized world requires a few more additional learning curves: applications in SMPTE ST 2110 as networked IP, virtual machines (VM), and not to be left out, security. That will be the broadcast engineer/network technician’s future, and that future is now!

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

In the many changes in media occurring during this age of digital transformation, production facilities are facing decisions about whether to go IP, while trying to determine the impact of producing content in UHD-TV1 (4K) and above.

In January, the 2019 Consumer Electronics Show featured evidence that 4K is here, stable and “readily” available—while looking squarely at 8K (UHD-TV2) as the next great change in television displays of the future. Of course, with 8K comes a need for creating and delivering that content—and, as the cycle continues, a quantum shift in how to efficiently manage the changes required to the infrastructure.

Obviously, the volume of bits needed to produce all this new content won’t really get any smaller. To add to it, the need to produce good content for 8K likely means it is shot in high dynamic range and at a minimum of 4K in resolution to realize the value of the larger screens, which will depend upon upscalers for quite some time.

Fig. 1 shows just how image sizes will relate to increases in bit volumes in order to meet the requirements to deliver high-quality video in the future.

MULTIDIRECTIONAL SCALABILITY

Throughout this past decade, applications associated with SaaS, AI, VR, social media and image resolution have demanded that systems scale in multiple dimensions. Peak demands continually change, forcing previously unseen growth in data sets and its management. New frameworks are needed to address these new data-intensive workloads in proportions that legacy solution sets cannot meet.

System resources are now being decoupled, allowing them to be scaled independently and turned into services versus how they were treated heretofore. Those who relied on the “shared storage” model are finding there is insufficient I/O (input-output) performance and excessive latency (i.e., throughput boundaries) to meet new demands, which goes for network interfaces and servers as well as storage.

Flash-based designs, now approaching more than 20 years since their introduction, are reaching a peak in terms of serial data I/O and transfers. No longer is it efficient to simply replace a spinning hard drive with an SSD and then tweak in performance on a legacy SAS or SATA interface.

Today, the latest up-and-coming technology is Non-Volatile Memory express (NVMe)—a scalable host controller interface coupled to a storage protocol that accelerates data transfers between client and/or enterprise systems that utilize high-speed PCIe (Peripheral Component Interconnect Express) based solid state drives (SSDs).

COMMANDS AND QUEUES

At least two primary factors affect SSD performance—commands and queues. A traditional SATA device will typically support up to 32 commands in a single queue; with the SAS device supporting up to 256 commands in a single queue. Here is briefly how these two constructs work:

Based upon the anticipated workload and system configuration, host software will create “queues” (positions or slots of availability)—up to the maximum supported by a controller. Often this is regulated by the core processor and is limited in number to avoid locking and to ensure the data structures are created per the cache of the core’s processor(s) without encumbrances.

A circular buffer, known as a Submission Queue (SQ) with a fixed slot size is used by the host to submit “commands” for execution by the controller. Each SQ entry is a command. A Completion Queue (CQ) is another circular buffer with a fixed slot size that is used to post the status for completed commands. The number of queues varies by the application—enterprise applications range from 16 to 128, with client queues only between 2 to 8. Block sizes for both are 4 KB (and above with NVMe protocols).

LANE CHANGES AND GIGATRANSFERS

Understanding the true meaning of the PCIe solution set can be complex, given the variety of interface opportunities and the interdependency of the I/O to and from the device being attached to the bus. Calculating PCIe bandwidth can be even more challenging, especially when giving rise to new terms such as Gigatransfers (GT/s) that are interchanged with rates and speeds, such as gigahertz (GHz). Rather than look at the explicit details of each “PCIe generation”—we’ll provide some introductory information about the evolution of PCIe in a simpler perspective.

Platforms utilizing PCIe connectivity have continued to rise, moving from Gen1 (24 lanes) to Gen2 (36 lanes with a doubling of bandwidth), to an I/O performance of 1GBps per lane in Gen3.

A lane is a data-transmission link, which consists of two pairs of wires—one pair for transmitting and one pair for receiving. Consumer PCIe slots can be 1, 4, 8 or 16 lanes. Packets of data move across the lane at a rate of 1 bit per cycle. The 1x link (one lane) carries 1 bit per cycle in each direction (hence two wires per direction times 2). A 2x link (two lanes) utilizes eight wires and transmits 2 bits at once (per cycle) in each direction. The numbers grow with successive PCIe generations.

PCIe Gen1.x and 2.x use 8b/10b encoding, which results in a 20% performance overhead. The encoding converts an 8-bit data set to a 10-bit character set, thus a per-lane 250 MBps bandwidth can only carry 200 MBps. When utilizing the gigatransfers parameter from the gigabytes bandwidth number, the numbers have a non-uniform change that is best presented in a table, which reflects the encoding, transfer and speeds for comparison (Fig. 2).

SSDs supporting Gen2 with eight lanes, deliver over 3 GBps; doubling that on Gen3 interfaces to 6 GBps for a single device. Encoding is now 128b/130b, resulting in only 1.5% overhead. Latency is also reduced and the ability to directly attach to the chipset or to a CPU was also revealed.

IT DOESN’T STOP THERE

AMD announced at the 2019 CES it would be the first to support PCIe 4.0 at either the enterprise or desktop (client) levels. Ironically, in May of this year, the PCIe 5.0 specification (providing four times more bandwidth than PCIe 3.0) was announced even before Gen4 (PCIe 4.0) was shipped.

Just how far will this go? Why does the development continue when, for example, Gen5 essentially has the bandwidth of a 100 GbE (Gigabit Ethernet) connection—equivalent of about 63 GB per second at 16x speed? Certainly, home users don’t need these values and even enterprise users might question this proposition—especially given the limits of the SSDs they would connect to and the costs of the switch gear—irrespective of the continually decreasing costs.

One evolving scenario for these exponential data rate growths is that of the “always-on” (vs. the “always-connected”) compute platform level, which is taking shape with the always-on model seeming to lead the race because of battery life expectations. For the media and entertainment industry, i.e., those routinely creating UHD/4K content, storage refreshes can now take less space, require less power and cooling, and increase performance multifold over previous storage solution sets.

EMERGING PROTOCOL

Current storage system advances are now based upon the latest NVMe protocol. Utilizing NVMe with multicore processors aids in removing bottlenecks that conventional interfaces alone experience. NVMe brings about highly scalable new capabilities for accessing storage media at high speeds, which in turn is enabling more growth for data-driven marketplaces including media, video production and post.

When you’re in the refresh mode or considering moving to UHD production, take a look at storage solutions incorporating NVMe, especially those who offer the repurposing of existing SSD, which you may already own.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Depending upon who you talk to or what you read, the topic of cloud seems to be one of the most prevalent of all the various technology opportunities—irrespective of whether the application is for science, research, media or data management. Cloud computing is listed by technology experts as “one of the topmost influential [IT] trends of this century;” lending credit to having fundamentally altered the way business and industry reacts to solving its technology needs.

The earliest known use of the term “cloud computing” (in print) is in a Powerpoint document for a business plan presentation at Compaq (founded in 1982) with the title “Internet Solutions Division Strategy for Cloud Computing.” That internal 1996 Compaq document stated that “Internet Cloud will have substantial impact on Compaq customers… across enterprises, small/medium businesses, and consumers and SOHO.” By October 2011, the term “cloud computing” had appeared 48 million times on the internet according to the MIT Technology Review.

Today, the cloud (a metaphor for the internet) and cloud computing impact a great deal more than just business and industry. With the emergence of the Internet of Things (IoT)—which is strongly connected to the cloud—use of the technology now affects aspects of both business and our personal lives in growing ways.

CLOUD, AI AND IOT

So, naturally, one of the higher-level topics of concern is the interaction of the cloud and the IoT, and how artificial intelligence (AI) will play into those activities.

Cloud computing, often simplified to “the cloud” involves the delivery of data, applications, media content (photos, videos, sound) and more to data centers utilizing the internet. The IoT, however, references the connection of devices to the internet and purposely segregates the computer or other straight communications devices—such as smartphones or computers (Fig. 1).

The IoT is generating an enormous amount of data, most of it in the cloud, and as such depends more and more on secondary intelligence (such as AI and ML) for its analysis, categorizing, storing or distributing. According to a two-year-old report from International Data Corp., “By 2019… 100% of IoT initiatives will be supported by AI capabilities.” This further infers that IoT and cloud will be directly connected, and intelligently managed, in order to sustain its growth and purpose long-term.

GROWTH OF CLOUD COMPUTING VS. THE DATA CENTER

Cloud generally provides multiple services; among those are the familiar offerings of compute, storage and disaster recovery—but the other offerings (SaaS, PaaS, etc.) are the “other side of the coin” with dynamics that are continually changing. For example, Infrastructure as a Service (IaaS) is, according to Gartner, forecasted to be the leader in the growth of cloud services, expected to increase in excess of 27% for 2019.

However, cloud application services, such as SaaS (software as a service) remains the largest segment of the cloud markets, revenue-wise, with growth around 18% expected between 2018 and 2019. Those providers who offer both IaaS and PaaS (platform as a service) will lead the pack when considering cloud-computing services for business and industry.

Observers and prognosticators alike seem to be aligning the perspective that, as the cloud matures and consumes market share, the typical data center will, in some circles, “die a slow death.” Perhaps the reality is that the makeup and functionality of the data center will adjust to balance the kinds of cloud services based upon applications, scale or purpose. With that, Gartner predicts that by 2025, 80% of companies will cease operations of the “traditional” data center and move those environments to that of a legacy “holding area.”

PRIVATE OR PULIC?

The argument for public vs. private cloud implementation also continues to ripple throughout the industry. Some believe and promote the private cloud, stating it can “decrease costs, increase efficiencies and offer more security” given that the private cloud lives within the firewalls of the organization. Datacenter operators who place computing resources and its hardware in a unified, software-defined and virtualized unit that is privately managed (for security or compliance reasons) are defined as “the private cloud.”

Effectively, the private cloud can provide similar services to those found in the public cloud, except that the private cloud’s abilities to scale are limited by physical space and capital availability; harmonized with the complexities of obtaining rapid change outside the boundaries of the physical datacenter’s walls.

The public cloud generally offers services that can scale to most any level, can provide flexibility of services “on demand,” but, like private cloud, offer certain challenges that counter the value of the on-premise capabilities of a private cloud.

MANAGED CLOUDS

Private clouds may be “in-house” (i.e., on-premises) or they may be provided by a private, third-party hosting service referred to as “externally hosted” or a “managed private cloud.” In-house clouds are run by the owner/organization, provide a greater sense of security and pose less risk to the organization in most cases. An externally managed private cloud, as opposed to the “public cloud” (such as AWS, Google, etc.), changes the security equation, issues of ownership and risk/worries over what happens should the provider be sold or go out of business.

A rapidly growing segment of cloud is that of the remote hosted private cloud. Here the organization has an ownership-interest in the physical hardware; it is just managed remotely by a third party. This practice was common during the mid-1990s when large co-located (“colo”) centers provided caged sections of the datacenter where customers would land their physical equipment and either care for it themselves or pay others to “keep it operational.” Such spaces were often placed directly adjacent to or in the same building where a major internet POP (point of presence) was located, thus providing quick access and less costly “last mile” services.

CONCERNS FOR CLOUD IMPLEMENTATION

One need not go far, on any search engine, to find the top five to 10 issues and concerns about this continually evolving cloud space. The top three-to-four concerns seem in near total alignment, regardless of which report or article you read.

The number one concern is that of security, often coined “the elephant in the room.” Ironically, until only recently (circa 2018), the number one challenge for cloud was the “lack of resources and expertise” needed to design, deploy and manage the services. Today, that has shifted, moving security to the top and the latter down a notch or two in the concerns stack (Fig. 2).

Depending upon where you put the expertise/resources availability concern, the next in the group of issues for cloud is that of cost containment and management.

It is generally agreed that cloud computing can save businesses money, but the long-term verdict remains open. Organizations don’t need to put huge investments into hardware that then must be recycled every three-to-five years as the obsolescence factor raises its ugly head. The ability, in a cloud environment, to easily ramp up processing capabilities without additional fixed cost investments in hardware continues to attract the customer to this environment. Pay-as-you-go models are further promoted by most public cloud providers; however, the predictability of user need (i.e., on-demand) together with scalability (i.e., growth balanced with requirements for services) sometimes makes it difficult to pre-define costs or predict actual total costs of operations.

Other risks to the cloud services environment include governance, control, compliance, performance and the rapidly changing capabilities of the various cloud service offerings that put a risk to engaging in any long-term contracts that may become stagnant compared to alternatives.

Some feel further that full scale applications should be spread across multiple clouds, driving the cost and deployment equations even higher. So is the answer still “build you own private cloud,” or do you go all in with the cloud and cross your fingers that, long term, you’ll fulfill your needs objectives and come out ahead in the overall race for success at the lowest cost? Time will tell.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Whether for television broadcast and video content creation, delivery or transport of streamed media, they all share a common element, that is the technology supporting this industry is moving rapidly, consistently and definitively toward software and networking. The movement isn’t new by any means; what now seems like ages ago, in the days where every implementation required customized software on a customized hardware platform has now changed to open platforms running with open-source solution sets often developed for open architectures and collectively created using cloud-based services.

These trends, concepts and methodologies are well into adoption, applicable to all media ecosystems, and extend to developments such as software-defined networking, storage virtualization, software-defined data centers (SDDC) and the like. The open concepts have even reached the storage domain in the form of software-defined storage (SDS).

TECHNOLOGY OR MARKETING?

Sometimes the buzzwords associated with these emerging trends can be marketing driven vs. technical fact or capabilities—once new terms show up in the industry, they seem to stick like the “technology” they are attached to. Take SDS for example—originally it was a marketing term for policy-based provisioning of computer data storage software. The management of that data’s storage is essentially independent of the fundamental hardware itself.

SDS hardware may or may not also have abstraction, pooling or even its own automation software. Whether for storage, compute or applications (apps), software-based products are driving techno-suppliers toward open-source development. With this comes new capabilities built on commodity servers in a virtualized architecture that provides “hands-free” routine operations, self-healing and automated load balancing, to name just a few.

Open-source development and implementation is happening across all segments of the manufacturing and media landscape; and continues to take a front seat in many compute and storage environments—especially in the cloud. One of the more recent open-source automation platforms gaining momentum is called “Kubernetes.” Originally designed by Google, Kubernetes is an open-source framework for automating deployment and managing applications in a containerized and clustered environment. Kubernetes is now maintained by the Cloud Native Computing Foundation (CNCF), founded in 2015 to promote containers.

Kubernetes, while closely associated with cloud-services, is not exclusive to the “public” cloud. Its principles and perspectives are applicable to cloud technologies irrespective of where that “cloud” is physically located.

To understand what this is about, we need to review some related terms: clusters, virtual machines and containers; noting these terminologies have perspectives beyond just applications—that the general principles are being applied to multiple operating environments including cloud, storage and other platforms, frameworks or architectures.

VMS, CONTAINERS & CLUSTERS

Virtual machines are nothing new but are becoming more prevalent as developers and users begin deploying applications (software) that run on multiple operating systems from the same hardware—especially servers. The compute power of modern servers allows several sets of tasks to occur simultaneously. These tasks must be managed, which occurs using software referred to as a “hypervisor” from manufacturers such as VMware or VirtualBox. As a process, the hypervisor separates the OS and the applications from the physical hardware; in effect insulating the tasks required of hardware systems (keyboards, displays, storage devices) from the core needs of the OS and apps (Fig. 1).

Characterized as “light weight, virtual machines,’ containers are standardized units of software that package up code and their dependencies such that applications can run efficiently (quickly and reliably) from one compute environment to another. Designers will create applications using containers, letting those applications be transportable to other operating systems, e.g., iOS to Android. A “light-weight” VM will share the machine’s OS system kernel, thus not requiring an OS per application. This promotes greater efficiencies, reduces server counts and decreases licensing costs.

Clusters are groups of (similar) “things” that are positioned—figuratively—close to each other. In computing, the coupling may be either “loosely” or “tightly” such that they are viewed as a single system to other resources. Clusters generally refer to servers—but for storage, this can relate to file system types, network-attached storage (NAS) grouping or to sector sizes on a disk (512-byte vs. 4-kibibyte or KiB).

KUBERNETES FRAMEWORK

At its core, Kubernetes is a framework, i.e., a means to enable automatic deployment with an ability to scale easily. It’s also about monitoring, a necessary subset that allows for the maintenance, notifications and for engaging self-healing techniques and modifications to facilitate updates, failovers and manageability during scaling.

Architecturally, Kubernetes has a master node that is part of a cluster along with multiple “worker” nodes (Fig. 2). Kubernetes knows about the other servers, which you can deploy containers to. Each worker node can handle multiple “pods,” with the pods containing multiple containers clustered as a working unit.

Designers start building applications using multiple pods; once complete, the system lets the master node know the definitions of the pods and number of pods to be deployed. Kubernetes then takes over and deploys the pods to the worker nodes unassisted. Once operational, should any of the worker nodes fail (go down), Kubernetes immediately deploys the pods to other functioning worker nodes.

Load balancing and the complex management of pods, nodes, etc., evaporates, letting designers focus on improvements and efficiencies at any scale required.

Originally deployed for cloud applications, Kubernetes is now being applied to other valuable operations whether as “cloud native” or “hybrid on-prem/in cloud” solution sets.

SCALING UP AND OUT

Kubernetes allows systems to scale, with the scaling functions automatically being assigned and managed. Generally, scalability (or scaling) refers to the strategies of adding more services or devices to maximize a target need, use or application. Scalability is characterized by two types—scaling up or scaling out. Scaling “up” means to add more resources to the same server or device; whereas scaling “out” implies the linking of lower-performance machines to collectively do the work of a more advanced single service machine. The latter is often thought of as synergy, where the sum of the parts solution is worth more than the numerical quantity of those parts by themselves.

Scaling up can be expensivem with some arguing there becomes a finite point where the value becomes less than the effort, due in part to the individual limits of the hardware (or storage) itself based upon performance. One example of those limits is when continuing to add small capacity storage (disks) to a cluster beyond the finite ability to control those devices or beyond their lowest common denominator—such as disk throughput or volume capacities.

Scaling out is often what’s behind bigdata initiatives. In the scale out model, a central data handling software management system administers enormous clusters of components (hardware), yielding flexibility and versatility.

STORAGE CLASSES

In Kubernetes, the “StorageClass” provides a way for administrators to describe the “classes” (i.e., the framework) of storage offered. These classes and their complements may map to quality-of-service levels or to backup policies. Other arbitrary policies determined by cluster administrators may be included.

StorageClass may be called “profiles” in other storage systems, but Kubernetes, itself, is moot about what classes they represent; its concepts and principles being insulated from the products it manages.

The Kubernetes web site shows the concepts, example code scripts, etc., which further describes how to utilize their resources.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

At this point in the technical evolution, we’re firmly in what might be called the “cloud computing era.” Yet, there is a somewhat interesting and possibly equally mysterious transition that is changing the location and the value proposition of the cloud.

Many of the new applications thought to be ideal for cloud computing may actually be occurring closer to the source of the data, that is at the “edge.”

At the personal level, many of us use products and services that are powered by intelligence that is found in the cloud. We place content in the cloud and we pull content from the cloud. These well-known products include those from Google (Chromecast), Amazon (Echo) and Apple (TV). Centralized services are familiar to many—those by Gmail, DropBox, Adobe Creative Cloud and Autodesk AutoCAD. These organizations use cloud services to store, backup, protect and interchange data and are always looking into new methods to achieve better security, decrease latency and reduce internet bandwidth traffic.

The “edge” is a relatively recent buzzword; similar in context to the cloud and the Internet of Things (IoT). Maybe “the edge” is not yet as familiar as AI and VR/AR, but it has similar importance to other technological capabilities that circle our universe and impact our lives on a daily basis. And people trust these services—we allow them to routinely collect our data and utilize it in ways that we can’t possibly manage ourselves. We let these companies “own” that data (unless you’re doing the GDRP thing), and we recognize that it’s literally impossible to halt the external/additional uses of our data due to myriad reasons, privileges or end-user-license-agreements (EULAs) that we graciously sign and accept before landing a single piece of data in their repository.

CLOUD DEPENDENCY

A significant number of companies have openly adopted the use of the cloud and rely on the hosting, machine language (ML), infrastructure and the power offered by the cloud. Nonetheless, the use of the cloud—as expected—continues to ebb and flow with the needs and capabilities of the technology ecosystem. Fundamentally, because of evolution, compute and other dependencies, we are beginning to see that certain functionalities and capabilities are now moving from the “public” cloud to what is known as “the edge.”

To see where this is headed, we start by looking at what “the edge” and “edge computing” is about. Basically, edge computing are those computational properties that are performed or “live” as close to the source or information gathering points of the data as is possible or practical.

But why do this when the cloud can do this work, probably faster and more efficiently than at the edge? There are multiple reasons and rationale for this, some influenced by technological evolution and others because the ecosystem, in whole or in part, allows this to happen.

PRIVACY

Families, institutions, enterprises, small businesses and individuals are increasingly more concerned about data privacy (and piracy). Many of you have likely had your credit card hacked at least once. While we may trust the cloud, and the firms providing the service—one can only wonder not if, but when something will be compromised.

If only the least amount and most applicable data needed is sent to the cloud service, then certain (unknown) risks might be mitigated. If only that data that absolutely needed to go to the cloud were encrypted locally, contained a biometric key and was sent over a protected channel—possibly using blockchain technologies—the user could feel better protected. To do this means that some of the compute process should be brought to the edge.

At least one major smartphone manufacturer does just that. They brought what was once only available as cloud-based computing out to the edge, creating a powerful change in how compute power is distributed.

SECURITY

We’ve heard about poorly managed IoT devices because the processing needed at the source was either absent or relegated to a location that passed through other less secure channels (e.g., the public internet) before reaching the cloud service-center destination. Today, browsers located at the edge, which have moved to the “evergreen” model, are finding success when employing edge computing principles in terms of increased security and better use of bandwidth.

“Evergreen” refers to services that are comprised of components that are always up to date. Evergreen IT encompasses services that are employed at the user level and at all the underlying infrastructures, whether localized, at the edge or in a cloud. Edge computing can help manage security by bringing only the needed information to the (public) cloud. Browser and cloud providers are working on OS and certified microcontrollers that will manage the types and depth of information that is cloud-bound.

BANDWIDTH

Proponents of edge services (for computing) believe that bandwidth can also be saved. That is, the bandwidth needed when to get every piece of data from host (user) to the cloud can be reduced if some or much of the compute efforts are done before going to the cloud. Artificial intelligence is helping to enable this. Say for example, in a camera security model, you are monitoring only one source (i.e., one camera) and carried only that full image to the cloud; there might not be much savings by edge “computing.” However, if you have several sites (i.e., multiple cameras), you could gain substantial bandwidth savings by combining the data—as smaller images from each of the individual cameras—into a single multiviewer and then transporting a single “composite” image to the cloud.

If you added AI at the edge to detect, for example, any movement—as in a motion detection security function—and only then switch the camera from one of several images on a screen to a single, full-screen size—then you’ve eliminated issues with having to continually look at every image as well as send only what is needed to the cloud for storage or other functions. Once any movement stops, the multiviewer reverts to all the cameras on a single raster. The AI function might also cache the other non-active images to a local (memory) store and hold it until reviewed by the user later.

LATENCY

Issues associated with bandwidth utilization may also be equated to minimizing latency. Major data information companies such as Google and Apple are working hard to localize the compute issues using AI to help control bandwidth and data-traffic demands. Furthermore, if you use “off-line first” techniques—i.e., you open the app on your mobile device without first connecting to the internet—then you conserve bandwidth without compromising performance.

Pay attention to these up and coming active changes when you consider IoT for your home security system or other monitoring features including thermostats, fire detection, etc. When appropriately managed, edge computing solutions aided by AI can help control security and return answers faster and more reliably.

Karl Paulsen is CTO atDiversifiedand a SMPTE Fellow. He is a frequent contributor toTV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

ALEXANDRIA, Va.—Broadcast facilities are already beginning the transition from SDI-infrastructures to IP-based network-centric facilities. Many expect significant changes to occur over the next five years and beyond. Content creation, production, and distribution entities will likely shift from the traditional SDI to an IP-based network topology employing common off-the-shelf (COTS) solution sets, steeped in software defined networking (SDN). With that change, the transport and manipulation of high bit rate (HBR), uncompressed (UC) media signals—especially for live/real-time production activities—will become the “next-generation infrastructure” of our future.

Recently adopted SMPTE standards, including ST 2022-6 & -7 and ST 2110; accompanied by industry forums (AIMS, VSF, AMWA) with their own initiatives and augmentations, are the driving forces that will essentially reshape the entire broadcast facility. Buckle your seat belts, we’re all in for an exciting and innovative ride.

RTP IP BRINGS NEW TIMING

In Part 1, we cited some “rules of engagement,” fundamental differences in flows, and how data traffic structures impact IT/IP broadcast system considerations. In Part 2, we will continue to examine how changing technologies will impact next-gen facility designs and implementations.

IP for professional media networks (PMNs) will add new “layers” to traditional live studio production systems. PMNs and the SMPTE standards supporting them utilize the Internet Engineering Task Force (IETF) Real-time Transport Protocol (RTP) for the purposes of timing, transport/identification and alignment of the packets in a Real-Time application. RFC 3550 (2003) is a memorandum which provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. It is a founding principle in the ST 2110 suite of standards and can be found at https://tools.ietf.org/html/rfc3550.

A new timing reference—known as Precision Time Protocol (PTP)—is based upon IEEE 1588:2008 (as PTPv2). PTP replaces the “video black” (and DARS) timing signals typically used in SDI, AES or analog systems. PTP uses a signal messaging system (Fig. 1) that determines, using propagation delay message exchanges, the precise time reference of the master for each slave device in the network. The PTP messages are sent to each network switch where they are passed on to sources (senders) and end-point devices (receivers) for packet timing and alignment. Each packet on a PMN will reference this common PTP signal timing via protocols described in the appropriate IETF RFCs.

PTP hierarchy consists of a Grand Master (usually with a backup) plus a series of boundary and/or transparent clocks distributed throughout the network (Fig. 2). A “best master clock algorithm” (BMCA) determines “who’s the boss” and “who are its minions”—allow for validation and prioritization in the event of a PTP generator failure.

Proper PTP system design is crucial to the system’s functionality and may vary based upon selected manufacturer’s products or system architectures. For broadcast applications, PTP is specifically refined and described in SMPTE ST 2059-1 and 2059-2. ST 2059-1, which set a point-in-time (i.e., the SMPTE epoch) reference which all devices are clocked from; and ST 2059- 2, describe how PTP works in broadcast centric applications.

PTP coordinates the timing for all audio, video and metadata packets allowing for system synchronization network wide, irrespective of the physical location of the equipment itself. This concept enables LANs to extend throughout a building, across a campus or between geographically separated environments.

PTP changes how devices are referenced on a network versus legacy black burst references in traditional SDI (and analog) video systems. Both PTP and video black reference may be implemented in next-generation facilities and are still quite common, given the hybrid nature (IP+SDI) of system designs.

LINE RATES, BANDWIDTH, FREQUENCIES AND FORMATS

While IP continues to evolve, SDI remains a robust and durable matrix-based X/Y routing and device I/O solution. SDI’s fixed bandwidths (270 Mb, 1.5 Gb, 3 Gb, 6 Gb, 12 Gb) and predictable performance enables isochronous video/audio switching against a reference point as described in SMPTE RP 168. Video router chipsets enable this switching for SD, HD or UHD signals accordingly. Although still quite usable and reliable, SDI employs a “fixed-matrix” that constrains facility growth bound to the fundamental I/O matrix (e.g., matrices of size 32x32 to 1152x1152 or above). Growing beyond a fixed-size SDI-matrix is complicated, expensive and may better warrant a forklift upgrade or the acceptance of a “blocking” architecture.

IP eliminates these factors. When designed properly, IP is essentially a non-blocking and unlimited- in-scale architecture. Fixed-matrices, prescribed bandwidths, and SDI constraints are eliminated in IP. With IP, the signal topology is constrained only by the port size and bandwidths on a given Ethernet switch. By employing spine-and-leaf switch architectures (Fig. 3) and sufficient uplink bandwidth from each leaf to each spine, the matrix-limitations of SDI are overcome.

However, this approach poses new challenges to designers; e.g., the initial switch selection must be thoroughly understood, be “PTP-compliant,” and be properly specified as most of the ports on each switch will run “full-tilt” and likely at a constant bit rate at or near the individual port’s bandwidth (i.e., 10G, 25G, 40G, 100G, etc.). Only properly configured media traffic will be permitted— i.e., no email, file-transfers or bursty/uncontrolled or unmanaged traffic should be injected onto this media network.

New systems are best designed to leverage the aggregate capabilities of 100G switches and beyond; utilizing an appropriate port-bandwidth to the signal format transported. For example, a single UHD (4K) signal will consume about 12 GB of data or “bandwidth,” making it impractical to deploy 10G-only ports on the network switches. For UHD using “native IP,” port architectures should be 25G (each)—permitting two UHDTV (2160p) signals per port in each direction.

The good news about IP: it is principally both format agnostic and bandwidth unconstrained (noting that 400G switches are just around the corner). To get more bandwidth, just add more leaf or spine switches. Format wise, running 1080p50 and 720p59.94 or UHD on the same switch is not an issue. If mixtures of 16:9 and 21:9 aspect ratios are anticipated, adding a 4:3 or even a 1:1 aspect signal isn’t a problem provided the sender’s and receiver’s devices accept those capabilities (noting that ST 2110 allows for picture widths and heights up to 32,767 pixels or rows).

IP essentially “future proofs” the broadcast signal transport agenda going forward. However, engineers must also be very aware of the internal architectures of the selected switch vendor’s products—things which must include PTP-compliant and possess the ability to manage the control and data planes through external SDN (software defined networking) principles.

NATIVE IP AND COTS EQUIPMENT

Probably IP’s biggest “promise” is that Common Off-the-Shelf (COTS) components will now do the heavy lifting. By deploying appropriate manufacturer’s switches, SFPs, and fiber optic media for transport and/or using general computing servers equipped with appropriate NICs, the ability to make the system extensible well into the future is achieved. Nonetheless, designers need to be aware of switch (ports) and server (NIC) capabilities when making choices—as many of these new devices may be only “partially-aware” or compliant with the emerging standards and protocols. Some broadcast equipment providers may insist on specific external server or gateway products which must be included to communicate with third party “broadcast controllers” or to manage signal flows when traversing the realm of multicast flows.

As time moves forward, vendor-specific sender/receiver components will move to “native IP” inputs and outputs. SDI may eventually become an option rather than the norm. Cameras, production switchers (vision mixers), audio devices, graphics and playout devices will—or already have—added IP-based interfaces. Instead of multiple BNCs or XLRs, devices will share common SFP-based I/O ports and include backup/secondary ports for resiliency (i.e., for ST 2022-7 or -8 topologies).

With an all-IP system, the need for SDI is reduced or removed entirely—adding flexibility to the system as it more easily expands. When the devices all interoperate properly—achieved, in part, by strict adherence to standards (SMPTE, IETF, etc.), the fundamental architecture of the system can be extended to meet growing demands long into the future.

In the final part of this series, we will explore the changes in cabling, a set of best practices and some human resource guidelines to prepare for the IP-transition.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

ALEXANDRIA, VA.—Broadcast facilities are now commencing what many believe will be a global transition from current digital (SDI) infrastructures to an all “IP-based (network)” facility. Not since the migration from analog to digital in the 1990s, has the industry experienced such a change.

On the surface, the transition seems logical, expected, and maybe even straightforward, given the level of IP/IT-integration already present at many facilities. Yet under the hood, both IT and broadcast technical professionals are in for a paradigm shift in concept, facility design and support practices.

This two-part article discusses the issues associated with next-generation IP-based facility design. The topics are not going to detail the transport of media over long distances nor the practices involved with file-based workflows, storage transfers or even OTT—which all use components of IP in an IT-domain. Instead, this article explores what broadcast and IT professionals will need to know about their future commitments to next-gen network-centric infrastructures.

CURRENT IP MEDIA PRACTICES

Professional media environments provide multiple means for moving video (i.e., compressed-files or streaming media) from point A to point B. Those points might be across the campus, between cities, to arbitrary distribution points—or anywhere between. In most applications, signal transport of the audio/video is a compressed video format of which there are dozens available. Some formats are highly compressed for Internet delivery and others are mildly compressed contribution- quality, e.g., from sporting venues to studios or for production integration prior to broadcast.

Manufacturers’ products for compressed media transport prepare the data for the subsequent stages in the content production chain. Few (if any), provide a means to transport uncompressed, high-bit rate content end-to-end over the network. This article doesn’t address the discussion for the absence of this form of transport.

With that preface, we’ll focus on the latest applications for high bandwidth, real-time, live broadcast production using IP over a media-centric network.

HIGH BIT RATE–UNCOMPRESSED VIDEO TRANSPORT

Broadcast production is beginning to use new capabilities for the transport and manipulation of high bit rate (HBR), uncompressed (UC) signals over an IP-network topology. These applications are specifically for live/real-time production activities. Recent SMPTE standards (ST 2022-6 & -7 and ST 2110 published year-end 2017), alongside industry forums and initiatives are driving new technological efforts that will reshape the broadcast facility.

“Internet Protocol” (IP) network technologies are already in use at many media facilities. The approaches are applicable to file-based workflows, data migration, storage and archive, automation and facility command-and-control. Previously, these weren’t necessarily called “IP.” Once the capabilities for UC/HBR video transport came about; that nomenclature evolved. Now, it seems, “everything” is IP, irrespective of how that terminology is applied to which application.

With that said, we’ll set the stage for what is happening in the future, and that ‘future’ is now.

RULES OF ENGAGEMENT—IT CHANGES EVERYTHING

IP is a “set of rules (“protocols”) which govern the format of the data sent over the internet.” For broadcast or studio facilities, “internet” is more appropriately called the “network.” Essentially, the application of certain constrained IP technologies will fundamentally address the facility infrastructure changes associated with studio/ live media-production and their content-chain processes going forward.

Designing and building an IP facility will require a renewed technological approach to IT-networking accompanied with a new mindset compared to those for traditional SDI-facilities. To comprehend what it takes to design, build and operate the IP-based professional media facility, an understanding of what “real-time” (RT) IP is and how it is differentiated from conventional SDI implementations (including file-based workflows or data storage) is necessary.

One key-target in this will be to keep “audio and video (over IP networks) acting precisely the way it does in an SDI-world” without the burdens or constraints of traditional SDI infrastructures. Fundamentally, facilities will leverage the advantages of network-based IP/IT structures for agility, flexibility, cost, and extensibility/expandability.

WHERE ARE THE DIFFERENCES?

SDI, born out of standards from the 1980s, was intended to permit transport and synchronously switch audio/video from source to destination without disturbances and to mitigate the generational quality issues associated with analog video and audio. The isochronous nature of SDI is straightforward for live, continuous video inside the studio and for long distance transport. However, these capabilities are not as easily accomplished in a file-based, non-real time, or streaming media environment.

Frame-accurate (undisturbed) transitions with compressed video, while somewhat possible in streaming media, is generally accomplished using peripheral equipment which essentially receives compressed video, then decompresses it to a “baseband” (SDI) form, where then seamless transitions from A-source to B-source are completed (Fig 1). Resulting signals may again be compressed to another format depending upon the application.

[Read: SDN: Not Just Another Three Letter Acronym]

These processes each take time, adding latency to the non-real-time chain. It is impractical for most live applications to cleanly switch sources and maintain timing and synchronization.

Program videos on YouTube or Netflix leverage sophisticated receiver buffering techniques or will make use of adaptive bit-rate (ABR) streaming functions to keep their “linear delivery” as seamless as possible to viewers. However, the ability to provide live and glitch-free source-by-source video possible is curtailed due to GOP (group of pictures) issues and compression/decompression latency.

For professional media IP systems—real-time/live signals, on a network, are transported over isolated, secondary or virtual networks (VLANs). For live and real-time, HBR signal transport, new network topology and timing rules must be adhered to. These “rules” (protocols) are defined in SMPTE ST 2110 and/or ST 2022 which include applications of IETF RFCs as defined in the new standards.

DIFFERING DATA TRAFFIC STRUCTURES

Another key point in understanding next-gen facility design is that differing data traffic types are not (generally) mixed on the same VLAN/network. Packet structure and formatting is different per each data type’s intended uses.

Real-time transport networks are conditioned to carry HBR traffic. Packets from senders (transmitters) are constructed based on IETF RFCs (Fig. 2) such as “real-time transport protocols” (RTP) and “session description protocols” (SDP); and supporting IEEE and SMPTE standards. Coupled with conditions identified in the SMPTE ST 2110 or ST 2022 standards— timing, synchronization, latency and flow control is managed so that the transport of media packets over professional media networks is possible.

One differentiator from previous IT-like network designs is that HBR traffic must run continuously at non-wavering data bit rates. File-based and streaming media is intended to, or can run at variable data rates. The data is likely to be randomly delivered and is often “bursty” in nature. In file-based transport, data from senders need not “arrive” at receiver input(s) in an isochronous (time bounded) nature. Streaming media acts in a similar fashion with fluctuating rates that are stabilized at the receiver end. Buffer sizes, connectivity bandwidth and variable file-data rates are accepted in these applications—but cannot be tolerated in real-time HBR applications.

In streaming media delivery, occasional interruptions or “buffering” is expected. That is a non-starter for live real-time video which must be synchronously time-aligned to allow for real time seamless switching.

Thus, a major difference in facility design is in how the various “network” segments are thought of. With that said, we’ll set the stage for what is happening in the future, and that “future” is now.

System designs now include distinct considerations for real-time and non-real time signal flows. Real time management and flow control associated with the endpoint peripheral devices must be “orchestrated” and will differ from non-real time delivery. File-transfer, storage and/or file-based workflows will likely reside on a different, less constrained network (segment).

Part one has now introduced broadcast and IT professionals to the differences and conditions associated with IP-centric professional media transport for studio and live operations. In part 2, we’ll discuss how next-gen design for IP-facilities differ and what engineers will need to know about their future in a multicast IP world.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor toTV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl atkpaulsen@diversifiedus.com.

Flash memory provides a needed level of storage performance for media and entertainment applications which now command, for UHD and beyond, massive amounts of storage and speed to meet the growing amounts of content being generated. New high-res applications demand much higher storage performance than conventional enterprise back office applications. Flash memory is helping enable that performance—but with it, the physical media also commands interfaces that provide the best value for the applications and its associated storage.

Non-volatile memory express (NVMe) is a trending technology that utilizes Flash storage more effectively and efficiently. The harmony of solid state and rotating storage media will continue for the near term, and for the undefined future. These respective storage mediums will endure, providing complimentary values for ambitions such as more storage, better storage, and faster throughput with a reduced hardware footprint, at less cost.

[Read: A Solid State of Non-Volatile Memory]

MORE THAN A 2X BUMP

At the previous two Flash Memory Summits (2016-17) both NVMe and the PCIe 4.0 bus were hot topics. Yet right alongside the 2016 Flash promotors, the SCSI Trade Association reminded the industry that “a new serial-attached SCSI (SAS) technology was on the way.” That new technology took the SAS ecosystem from the 12G level to a usable 24G SAS (24GBps, serial attached SCSI). Promoters unveiled this advancement as more than just a “two times bump in speed over the previous data rates for 12G SAS” (refer to Fig. 1 for the SAS technology roadmap).

The interface of choice for mission-critical storage applications remains SAS; moving the interface from 12GB to 24GB yielded a major refurbishment in the technology. Updates included more efficient 128b/150b encoding, with SAS Protocol Layer (SPL) packets and Forward Error Correction (FEC). The transmission signaling rate is specified at 2.4GBbs (i.e., 22.5 gigabaud rate) which still retains compatibility with earlier 6G and 12G solutions.

Changes which help achieve this faster throughput include an FEC field, 20 bits in the SPL packet that aid in error detection and recovery. An SPL packet is a 150-bit block that includes a 2-bit header and a 128-bit packet payload, plus the FEC bits. At the deep-dive level, additional features added include changes in: binary primitives (in the SAS Link Layer); primitive parameters; serial management protocol (SMP); open priority; and inter-expander fairness arbitration enhancements. Details can be found in a technical overview document for Serial Attached SCSI, which is roughly 1,000 pages and whose details are beyond the scope of this overview.

LOWERING LATENCY

Regardless of the application for media and entertainment or others such as transactional trading on the stock exchange, latency has a tremendous impact on storage performance. Recalling from our previous article—NVMe is an industry standard that is optimized with a new storage stack featuring a low latency, efficient and scalable protocol streamlined with a revised drive command set that uses fewer clock cycles per IO operation.

Comparing spinning magnetic drive latencies for hard disk drives (HDD) to those of Flash SSDs, conventional Flash NAND technology offers a 100x reduction in latency over HSDs.

Latency figures decrease further when NVMe eliminates the 20 microseconds of latency found in the SSD NAND (whether SAS or SATA) implementation. “Next Gen NVMe” will now drive NVMe to deliver “4KB operations in under 10 microseconds,” according to a presentation at the 2016 Flash Memory Summit.

CHANGING THE MESSAGING

What do these memory improvements provide to the media and entertainment (M&E) industry? For starters, it enables audio/video to become the dominant medium for the carriage of information and content. In a February “New York Times” text message to mobile subscribers we saw, “What you are doing now (i.e., ‘reading text on a screen’) is likely going out of style.” The Times’ text infers that A/V will replace the text messaging we do today and in the not-too-distant future.

One of the tasks necessary to achieve this change will be to increase the speed of the delivery while at the same time improving the processing and memory requirements of the channels which deliver the information. The new 5G networks may help this, but more is needed. This same prolog can be applied to production and post-production workflows for M&E.

[Read: Non-Volatile Memory Grows In Popularity]

We have already seen the impacts of 4K over HD in terms of resolution, quality and image perception. While well-produced 1080p content can be wonderfully upscaled for 4K displays; in many cases there can be very satisfying results when the content is originated as 4K and then downconverted to 1080p and displayed on a high-end HD (1080p60) television system.

The real impact for UHD/4K, once available in more delivery systems, will be in the use of higher frame rates (1/120 second frames instead of 1/60 second) especially for sports. Add the higher resolution per frame and a higher density of pixels per unit screen area, and it could make today’s conventional HD images look like older analog 1-inch videotape, comparatively.

IT TAKS A LOT OF MEMORY

To reach these goals, it takes memory, a lot of it; and fast memory coupled with much higher bandwidth (selected video formats described in Fig. 1). Compression helps to a degree, but with the improvements in compute and networking technologies, more productions are switching to full-bandwidth, uncompressed video for their editing and post-production workflows.

Digital storage for media workflows will continue to advance in these areas. Flash, coupled into NVMe interfaces, make these higher capacity, faster and better memory solutions possible. We’ve seen Seagate produce a 12TB “BarraCuda Pro” 3.5-inch HDD with SSD-like performance, and SanDisk’s “Extreme PRO” CFast 2.0 solution in a 256GB form factor capable of 525MBps read (450 MBps write). These devices are the tip of the iceberg, as the products become the workstation’s local drives (i.e., the Seagate HDD) and the camera capture’s memory solution (i.e., the SanDisk memory)—key components needed to work in higher resolution and higher frame rate production workflows.

ACHIEVING THE TARGET

Along with this perspective on higher resolution, higher bandwidth production—another element which will consume massive amounts of memory is video on demand. According to Coughlin Associates, the shipping capacity for VOD storage will increase 8-times, from the 2,500,000 TB (terabytes) in 2016 to a whopping 20,000,000 TB (per year) in 2021. Add the other data sets, which aren’t getting any smaller, and the requirements necessary to fulfill all these ambitions is quite an undertaking. When AI and VR take hold, cloud services may not be enough to satisfy such exponential changes in storage volumes—let alone the means to transmit (wirelessly) the demands to the end users.

To manage the physical maintenance issues of storage using HDDs could require small armies of support teams (quite possibly robotic) just to change the drives in even a modest data center. Flash memory would need less support, with the R&R (rescue and recovery) cycles for SSD upkeep probably hundreds of times less. Technological changes would likely outpace the life of the SSDs in service, making storage a disposable commodity.

This is the future for storage. Mechanical drives, besides reaching physical capacity thresholds that may limit their overall performance, are probably in their last decade of useful life—at least for large scale M&E solutions. Single unit workstations may continue with HDDs, where cost-to-capacity requirements are different than in datacenters—although the migration to all SSDs is very much apparent in tablets and mobile devices. Watch for some dramatic changes in storage and Flash as more video enters into our lives and better images become an expected case and not a “luxury once in a while” alternative.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

No one goes to an event like the annual NAB Show without doing their homework first. As the show expands its umbrella to cover more advanced media technologies, the need to prepare becomes ever more crucial. Whatever your taste, TV Tech’s experts are here to help; here’s their advice for 2018:

KARL PAULSENStorage Technology

Standout trends will likely center on evolving workflows in cloud-based solutions and emerging applications for IP infrastructures. This is the first NAB since the adoption of new SMPTE ST 2110 standards for Professional Media Networks so don’t miss the IP Showcase (in the rear of Central Hall) where working examples of the new standards plus integration of the NMOS interface specifications will be shown in an educational showcase environment. Potential IP adopters will be looking at how manufacturers address software defined networking and new tools aimed at diagnostics and operational management for IP implementations.

The enormous prominence of virtual and augmented reality and artificial intelligence and machine learning at the Consumer Electronics Show will be evident at NAB. Expect to see evolving products necessary to support the industry’s mandate to create, manage and deliver content to these emerging platforms. eSports is now attracting inventive players and changing production techniques that may show promise for aspiring venues.

The continual industry churn of what is now Belden’s growing empire should attract users to see what new products they collectively offer. Everyone is curious how the combined companies of Grass Valley and Snell Advance Media will address the changes in infrastructures that appear to be moving away from pure hardware and into virtualized, software-based environments. We’ll see what comes out of these mergers and acquisitions—and who will be next in line.

JULIA SWAINLighting Technology

The 2018 NAB Show promises to be a big one! Excited to see more lighting units capable of RGB and DMX, which so many of us have been utilizing more and more on set. Being able to move so quickly between colors and qualities of light has opened up a lot of possibilities. I’m very much hoping for LEDs with great outputs as well. The climb toward stronger, more versatile LED units has been an exciting and consistent one, so I’m looking forward to seeing what this year brings in the world of lighting.

On the camera side, I anticipate some new monitor options with a gamut of exposure tools. Lots of apps to control and learn camera settings are also on the horizon.

AL KOVALICKCloudspotters Journal

Look for all things cloud including SaaS apps for your daily operations. Don’t settle for installed apps unless there is a performance need. Ask vendors what their cloud strategy is, including what clouds they support for media services, apps and processing. Go to NAB with a list of “cloud questions” for your preferred vendors specifically around hybrid cloud local operations integrated with cloud services. Understand there is a place for local services but these are being eclipsed by cloud operations. Understand what mix will work best for your facility. Expect to use one or more clouds to meet your business needs. This multicloud approach will give you more flexibility for business operations. Look for 24x7 cloud support and operational services possibly from specialty companies.

JAY YEARYFocus On Audio

This is the year where we really start to grasp the full scope of the changes that IP-enabled technologies are bringing to television, from ingest all the way to delivery. HD-SDI video and discrete audio chains will see fewer implementations as they are passed over for IP-based alternatives, even though the road to an all-IP facility remains a bumpy one. With ATSC 3.0 now rolling out in the U.S., IP is now a reality for new and remodeled television facilities.

In audio, we’ll certainly see an increasing number of personalization options for consumers, along with products for immersive audio that are designed to be shoehorned into residential environments. At least for now however, it looks like personalization, whether mono, stereo, or emulated surround, has piqued the interest of the end user more than additional surround channels in the living room. This could change if the costs of immersive audio products for the consumer become a little more accessible. The preference for personalization is partly VR-driven but is really a continued outgrowth of the de-cades-old personal device boom—which is likely to continue with or without a VR element.

User interfaces for Next Gen technologies are particularly worthy of scrutiny this year, since presenting complicated options in an easy-to-understand package is an art form that will make the difference between success and failure for some products. Finally, anyone hoping to stretch out their use of 600 MHz wireless devices appears to be out of luck now that T-Mobile has accelerated their rollout.

CRAIG JOHNSTONCorrespondent

We’ll see the rollout of ready to use 4K and some 8K camera systems. These new cameras have spawned attendant equipment such as enhanced lenses and high bit rate signal transmission equipment.

A whole host of 360 degree virtual reality camera systems and stitching software will be presented. An Immersive Storytelling Pavilion will help newcomers to 360 degree technology figure out how it will fit into their business.

Look for cellular liveshot gear that is futureproofed by including 5G capabilities, even though 5G at present is a small blip on the cellular radar.

And speaking of futureproofing, a lot of black boxes being bought at the show will have IP connectors on them, even though they will initially be connected via coaxial and fiber optic cable.

Click on the Image to Enlarge
Click on the Image to Enlarge

Even as a mature, diverse and reliable technology—magnetic spinning disk drives (aka hard disk drives or “HDDs”) continue to grow in capacity, performance and cost benefits. Alternative storage solutions, however, continue to evolve. Hot on the HDD heels—as has been the case for more than 10 years running—are a family of solid-state equivalents.

SSDs (solid-state drives) now sit squarely alongside the other legacy nonvolatile memory solutions, and their presence is being enhanced by interface improvements known generically as non-volatile memory express or NVMe. 

Storage media itself, such as Flash Memory, have seen a multitude of improvements centered on many technological advances. My previous columns have outlined those changes over the past decade. Now we see new steps to improving non-volatile memory solutions, which are in the interfaces themselves.

HDD EVOLUTION

Of the various interface forms for HDDs, those with serial Advanced Technology Attachment (SATA) interfaces

have become the more cost-effective and most prominent of the “everyday application” disk drives. The more expensive serial-attached SCSI (SAS) drives buy the users other capabilities.

When it comes to HDD applications, SAS drives tend to be found more in enterprise computing because of their high speed and high availability, factors crucial for such activities as ATM transactions, stock exchanges and eCommerce. 

Conversely, SATA drives are used primarily in desktops for consumer use and in those less demanding roles such as backups and near-line data storage.

We are omitting Fibre Channel disk drives from this conversation because they are more specialized and less cost effective, but can arguably be justified in high-performance storage applications (such as editing systems) when supported by the appropriate operating and file system technologies.

IOPS AND RELIABILITY

Keep in mind that the best measure for HDD speed is IOPS (inputs/outputs per second), specifically when the drives are in use, under stress and with real applications designed to optimize the drives’ capabilities.

To put IOPS into perspective, industry-accepted averages for 7.2K SATA drives is about 80 IOPS, with the 10K (RPM) devices offering around 120 IOPS and 15K pushing the limits of around 180 IOPS. The equations turn dramatically for solid-state storage devices (SSS), with huge IOPS improvements and no mechanical worries. 

The other factor for SAS v. SATA is reliability. The industry-acknowledged MTBF (Mean Time Between Failure) for SAS HDDs is around 1.2 million hours, compared with 700,000 hours MTBF for SATA drives. This also places SAS clearly into the enterprise space.

Once you move from the HDD world to the SSD world and to other forms of non-volatile memory, the feature sets and interfaces begin to shift. Not only will speed (performance) increase, but the applications for which SSDs can be applied to help increase total system performance—not just from the storage I/O perspective.

Table 1: Some familiar, but less recognized non-volatile memory (NVM) types.

Non-volatile memory (NVM), sometimes called NVS (non-volatile storage), is a classification for a form of digital storage (memory) that retains its state without having power continually applied. Generally, this storage media is without any mechanical components, although that is not necessarily the case.

Optical storage is considered in the NVM classification, as would be any readonly storage that doesn’t require electrical stimulus to retain its state (e.g., a PROM/EPROM). See Table 1 for examples.

Initially, NVS and NVM were intended for secondary storage or for other longterm persistent storage mediums. Today, in its SSD format, it is often used for primary storage to support short-term RAM/DRAM—as in laptops, tablets or mobile devices.

INTERFACE EXPRESS

Besides the physical storage media, electrical and software interfaces are needed to support the “NVM-storage” term. Common interface methodologies include Non-Volatile Memory Express (NVM Express or NVMe) and NVMe over Fabrics (NVMe-oF or NVMeOF). See Fig. 1 for an example of one method for the physical interface.

The “NVMe” term mystifies many, with some feeling these terms are becoming more hype than practicality. We hope to provide some clarification with the following.

NVMe is a host controller interface and storage protocol established to accelerate the data transfer between host/enterprise or client systems and solid-state drives over a computer’s high-speed PCIe (Peripheral Component Interconnect Express) bus. NVM Express (v1.3) is an open collection of standards and information that exposes the benefits of nonvolatile memory (NVM) in computing environments from the mobile device to the data center. The specification, and its registered and trademarked explanations, can be downloaded from the NVM Express Inc. website (www.nvmexpress.org). The term “NVMe” is a trademarked name, which encompasses a solution set designed, from the ground up, to deliver high bandwidth and low-latency storage access for NVM technologies.

The NVMe specification defines a register interface, command set and collection of features for PCIe-based SSDs. Its goals are to enable high performance and interoperability across a broad range of NVM subsystems. Note that, like most “standards,” the NVMe specification does not stipulate the ultimate usage model, i.e., how NVMe is directly associated with a specific solid-state storage, main memory, cache memory or backup memory. 

NVMe is optimized for Enterprise and Client solid state drives, typically attached as a register-level interface to the PCI Express interface. The 287-page specification describes an interface that allows host software to communicate with a nonvolatile memory subsystem. During its development, the specification was referred to as “Enterprise Non-Volatile Memory Host Controller Interface Specification (NVMHCI).” The lengthy name was simplified to NVM Express prior to publication.

Fig. 1: PCIexpress interface card with NVMexpress solid-state drive unit fitted to slot on the interface card. Card uses an M-Key edge socket to attach the SSD to the PCIe form-factor adapter card.

The NVMe 1.3a spec addresses both NVMe over PCIe and NVMe over Fabrics. The later Fabrics specification defines a protocol interface and related extensions to NVMe that enable operation over other interconnects (e.g., Ethernet, InfiniBand, Fibre Channel). Support requirements for features and functionality may differ between NVMe over PCIe and NVMe over Fabrics, rendering different performance parameters based upon the interface application.

NVMe increases support for Enterprise capabilities via enhanced error reporting and virtualization. End-to-end data protection is compatible with SCSI Protection Information, known as “Data Integrity Field” (DIF). DIF (or T10 DIF) is an approach to protect data integrity in computer data storage from data corruption, originally proposed in 2003 by the T10 subcommittee of the International Committee for Information Technology Standards (INCITS). SNIA references this as “Data Integrity Extension” (DIX) in its standards.

KEY FEATURES

The NVMe 1.3a spec interface has many additional key attributes, such as an efficient and streamlined command

set and support for multiple namespaces and namespace sharing. The NVMe Management Interface is the command set and architecture utilized in out-of-band management of NVM Express storage (e.g., discovering, monitoring and updating NVMe devices using a BMC). Typically, the NVM Express controller is associated

with a single PCI function.

In similar fashion to the transitions of HDD interfaces that went through ATA, IDE, SCSI and beyond, now, when you think about solid-state drive technologies, you can add the latest dimensions of the NVMe interface—one of the hotter topics in storage technologies that are growing stronger each day.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. Read more about storage topics in his book “Moving Media Storage Technologies.” He can be reached at kpaulsen@diversifiedus.com.

How many times have you written a file to a directory only to find out, sometimes much later, it was corrupted, lost or improperly archived? None of us are likely immune to this phenomenon and few are aware of how this might be prevented.

The accuracy, quality and consistency of data—regardless of where it is stored—is of immense importance; yet data integrity is often taken for granted by most users. Irrespective of where your data is stored—whether in a warehouse, data mart or some other construct, including the cloud—guaranteeing data integrity may be the most vital parameter in the entire compute chain. 

Data integrity describes a state, a process or a function. It is often used as a proxy for what is sometimes referred to as “data quality.” Data integrity is routinely equated with “databases,” as in “ensuring database data integrity” but it can a mean a lot more, especially when referring to the many actions that might or can occur in manipulating files (data) through various workflows and processes.

Fig. 1: Trusted values for data integrity.

Data integrity is warranted whenever the original data is modified, as in the editing, copying or transferring processes. Another concern is when the data is backed up or archived. The methods practiced by software and hardware storage solution providers need to be “transparently trusted” so that errors or losses of data are prevented. When or if data errors are detected, they should essentially be unnoticed (i.e., “transparent”) and never impact the results of when that data is used for computational, display or delivery purposes.

TRUSTED DATA

Data integrity infers that the data is trusted; that is, the data properties are trustworthy from a technical perspective (Fig. 1). Data must be trusted by the consumer of the data using reliable reports on the data’s state or status, as well as with the applications that will use the data.

Trusted data includes such attributes as “complete data.” This is achieved when the data integration technologies and techniques produce a consolidated data structure. The term “enterprise data warehouse” (EDW)—a term applied well ahead of today’s “cloud services” world—is where organizations could place their data (near term, short term or deep/long term), as a secondary service to on-premises (“on-prem”) storage. Respected and trusted EDWs should be competent enough to provide its customers with a full 360-degree view of users’ data with a historic context of all the real-time data activities. Cloud storage services have effectively replaced the legendary EDWs; with functions and expectations that are essentially the same.

STALE OF CURRENT

Trusted data should be “current data.” Users should be able to query the data system to understand questions such as “how old is the data” via a report?” Trustworthy data is “fresh,” whereas “stale data” is not. Stale data can be contaminated by successive read/write processes (where data is copied or relocated during defragmentation) or by “bit rot,” the slow deterioration in performance or integrity due to issues with the storage medium itself. 

Existing data may be corrupted when there is considerable activity (multiple seeks, reads and/or rewrites) on the actual disk platter or across the physical tape medium due to successive passes and/or writes. On optical drives, if the physical areas of the disc where the current data resides, has been untouched (unwritten to) for considerable time and then the laser writes to an adjacent area (or surrounding tracks); there are opportunities to blur or distort the existing pits making your existing data unrecognizable or questionable.

CONSISTENT AND CLEAN DATA

Another data trust factor deals with supporting the other (i.e., meta) data associated with the main “core” data. Properly maintained metadata management and master data management practices help to ensure that the data you need can be searched, found and retrieved on a consistent and secure basis. Good metadata management includes documenting the data’s origins and meanings, in a reliable and coherent methodology. 

Employing data quality techniques is critical to obtaining “clean data.” Typically, this is obtained by following standardizations in data management, using data verification and matching techniques, and using data deduplication. To maintain operational excellence and to make quality decisions, your data must be clean.

Since data activities often include the aggregation of one data set with other data sets as data traverses workflows, or when data travels across multiple IT systems; it is important to know and trust that the data is technically sound and consistent across the enterprise.

COMPLIANCE AND COLLABORATION

Ensuring you have “regulation-compliant data” is another element of a trusted data scenario. Regulations associated with data compliance come from sources which may be external to your organization, as in federal legislation or your partner connections. They may also come from internal policies such as your own internal IT data architectures, quality assurance, security and privacy. Businesses need to trust that their data has been accessed and distributed in accordance with both external and internal guidelines. 

Data “collaboration” (also referred to as data “sharing”) is essential to business functionality. Collaboration helps to ensure a strong alignment between data management practices and business management goals. When successful, data collaboration improves trust amongst inter-departmental activities and functions.

Fig. 2: Data quality monitoring, attributes to data integrity.

SOURCE TO DATA INTEGRITY

Data integrity needs to be maximized from the data source to the application and through to the storage medium (disk, tape, optical or cloud). Data integrity (and protection) is made possible by applying multiple mechanisms during the write processes, modification processes and through the management of duplicated data throughout the system (Fig. 2). 

One methodology is to ensure that when original data, stored for example on a disk drive is modified, that the results will always be written to unused blocks on that disk. This practice assures that the old (previously unmodified) data is unaffected on the disk, even if the new data written to the other location is corrupt. Through tracking history, if one data write is found to be in error, you could fall back to a previously written block and recover either the same (or a previous) version of the disk.

Another level of data insurance, the “snap shot”—from the photography term—is the capture (recording) of the state of a system at a particular point in time. Besides protection, this read-only snap shot of data can also be used to avoid downtime. In high-availability systems, a data backup may be performed directly from the snapshot. This method lets applications continue writing to the main data while a duplicate “snap shot” of data becomes that data, which is backed up to another storage resource.

There are many secondary (provided by the storage vendor) and third-party (non-storage vendor) software solutions that can manage both snap shots and backups in real-time.

INTELLIGENT BACKUP

As expected, more organizations are now considering modern intelligent backup practices to protect their data. This means more storage is consumed and that more data system management will be necessary. Intelligent backup solutions may utilize a practice called “deduplication,” which finds all the instances of any data set, establishes pointers to where all those instances occurred relative to the data backup, and then narrows the amount of data down to only a single set of data. Deduplication reduces storage costs, but also increases performance of data backups and restores. 

Data “dedupe” (its shortened name) can be used on main data (e.g., in databases or on transactional data sets) or for backups or archives. However, due to the already highly compressed nature of video along with the unstructured nature of motion imagining data, deduplication does not improve the storage performance in the same was it would for structured data.

Data management technologies, monitoring, and practices continue to evolve. How data is managed on a solid-state storage medium (SSD) versus a magnetic spinning disk or optical disc can vary. Users looking into new archive platforms or backup solutions should absolutely get the full story on how the vendor handles data management, data integrity and resiliency—before making a selection that they will live with for years to come.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl at kpaulsen@diversifiedus.com.

In my last column, we looked at the fundamentals of object storage as it applies to the media industry and as applicable to archiving. Object stores have become a principle solution for long-term data preservation, especially for cloud-based environments—whether for on-prem or private clouds. Object storage is used heavily in public cloud storage solutions and especially when the data is geographically disbursed for protection and accessibility purposes.

Furthermore, where tiered storage has been a trend for more than a decade, object storage brings new perspectives—particularly when addressing disk replacement management. Object storage may now be overshadowing even some of the earlier approaches to tiered storage. 

A comparison of how traditional versus object storage is implemented

Tiered stores were, for many years, an integrated overall approach to having different drives and physical media applications arranged in such a way that high-performance drives (those at the highest/top tier) were used for immediate access by applications; and lesser performing storage solutions (e.g., tape) were used for longer-term storage.

HIGH-PERFORMANCE, TOP-TIER STORAGE

Applications, for example those for post-production video editing or graphics compositing, require data delivery to be extremely fast and do so with minimal latency. Often these top-tier storage devices were Fibre Channel based (i.e., both the drives and the network switch configurations are Fibre Channel). These systems would be coupled with specific file-system controllers (aka “metadata” servers) which are designed to manage the high-volume/high-throughput transfers necessary for very rapid data movement to and from the base system application servers, local workstations or the compositing, rendering or effects servers.

The mid-tier storage was often referred to as “slow-disk” or “near-line” storage. The content data held at this level were usually the completed works (finished edits), as well as complimentary versions of the first master completed clips, stories or commercials. In addition, various “b-roll” pieces might be held in the same near-line storage since fast accessibility was not necessarily needed during content approval periods or lulls during the post-production periods. Often this storage tier was also used as the holding location for content which would ultimately be readied for archive or much deeper/long term storage.

The drives needed in this application need not be expensive, top-tier level (fibre channel) drives. Even the types of drives used in consumer-level PCs or laptops would suffice for this level of storage.

ARCHIVE NEEDS AND CONSIDERATIONS

The lowest tier storage was usually a linear tape storage solution or possibly cloud storage. If stored to tape, sometimes the workflows would require a local copy in a tape library be retained, and an off-site copy (e.g., an “iron mountain” version) would be created and physically shipped to another location.

It is this lowest tier, the archive tier, which is steadily being replaced by the disk-based object storage solution. The reasons for using object storage vary depending upon the current investment the archive-provider may already have in tape-based storage; or how the organization feels about shifting to newer technologies (i.e., disk storage either on-prem or offsite); or if the organization feels tape storage is too costly or they’ve never used tape-based library solutions and don’t want to pay for the costs of private or public cloud for archive.

In the latter case of cloud storage costs, this rationale will depend upon what form of the cloud you choose or if the potential time line needs for data restoration (recovery) cannot be predicted. These factors change the equation in terms of both the cost for data recovery (i.e., getting the data back to your own mid-tier storage) or how quickly, time wise, you might expect that data be needed since some archives use storage which cannot be easily just called up and returned to the user.

Deep archives are meant to take data in at very low costs and charge you much more to return that data. The more quickly you want the data recovered, or the volume of the data you need plus the time you need it back can seriously impact the costs for storing (and recovering) that data. 

FAST DATA RECOVERY IMPACTS

Some organizations simply don’t know when they’ll ever want the data back or how fast (e.g., disaster recovery versus occasional return uses). For this reason, by itself, building an archive which allows for the rapid return of the data (whether from offsite or onsite tape or from the cloud) becomes a business-level trigger whereby you might want to consider object storage and exclude tape or cloud, except in a few situations.

Another capability inherent in object storage-based solutions is the cataloging methodologies utilized in the store itself. Metadata is key to search and paramount to getting just the right data back from any store. In most non-object cases, the metadata is held in an external media asset management system, which needs its own database, application servers, and integration into the entire storage and workflow solution set. 

When the key metadata can be held within the object (store) itself and be searchable or modifiable by an external solution (that is, the object’s indexer), then the necessity for an archive-centric MAM changes. In objects, which are essentially “wrappers” or “containers” that hold both the content data and the rich metadata together as a single entity; only keywords and locations need to be addressed by these external MAM-like applications. This reduces hardware and software overhead, and allows the systems to be rapidly accessible. In many cases the object store solution will take less physical space (a smaller footprint) compared to the tape library and cassette storage needed in more traditional archive applications.

LONG-TERM COSTS AND BUDGETS

While this point might be arguable depending upon when, what or how much the user has invested in a tape solution—the long-term costs for object storage, once implemented, can be reduced versus tape. Why? Because tape has a finite life (both physically and technically) that mandates the data be migrated from an older format to more current higher density formats. If you have a large library, this means the older tapes (which hold far less data) are or will need to be updated; that is, replaced with the most current solutions on a recurring basis. This replace and renewal process may be every few years. So, both the physical media (tapes) are replaced and the tape-drive mechanisms will also need replacement to leverage the newer, higher density/higher capacity tape storage solution. And don’t forget, if the physical media is damaged, the data is lost forever, unless a second copy is held elsewhere.

In object storage solutions, there will be multiple drives which (in similar fashion to RAID) protect the data across many sets of drives of that object solution. In some object store chassis, which may for example, consist of 20 to 40+ drives in each group set, as many as 4 to 10 drives can fail—and all the data can still be recovered. In large data centers (or clouds) disk drives are seldom replaced since the labor to remove and replace a single drive—per instance of failure—cannot be justified. When an object group gets to the point it becomes too high of a risk to implement data recovery (that is, maybe 6 of the 8 drives in a 20-drive group have failed); a decision can be made to update that chassis, or maybe retire the chassis altogether. 

MAKING BEST SENSE

Object storage makes sense when rapid access to data is necessary; when the risk to not being able to recover the data quickly is unwarranted for your operations; or if a cloud solution is not financially beneficial (coupled with either of the other two reasons). And since object storage systems allow you to replace older smaller drives (e.g., 2 or 3 TB drives) with higher capacity drives (mixing multiple size drives in the same group set or chassis), then the migration issues associated with tape vanish.

Next time you need to look at archiving content without the complications of tape or cloud, consider how an object-based system might fit into your organization’s workflows and budget.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl at kpaulsen@diversifiedus.com.

For some time we’ve thought mainly about how file-based storage is used to contain unstructured data; i.e., those files relative to moving (video) or static (photographic) images. File-based storage has traditionally been suitable for structured data—such as computer information when employed on personal computers or office workstations. This type of data can be appropriately organized, that is “structured,” for applications such as email, discrete sets of “office” documents, and project-related applications.

As “media”-related data came into the workforce, the storage of that data was also managed in the same way, despite the absence of true organization (or “structure”) to that data and no “best practices” for how to manage it. This type of media-centric data became known as “unstructured” data; a term which lives on through today.

In the early days of digital graphics (generated as individual files), photographic images (also individual files) or video (as contiguous groupings of inter-related files), few realized that digital media would expand to the degree it has in the past decade-plus years. Little work was done to address this eventual quantum shift in data storage needs and requirements. As linked sets of JPEG images for professional video media moved to streamed “strings” of compressed data, the volumes of files continued to grow in terms of both the formats and the quantities of actual content.

There were those who believed that asset management solutions might provide the needed organization of unstructured data. Those MAM-like processes however would still rely on traditional file-based storage solutions on the physical media (tape, hard drives, etc.), with content driving many new features and functions. Harddisk capacities increased, metadata became more important, better caching methodologies were developed, and transfer bandwidths grew to address the speeds and file-sizes of this new era.

When content creation exploded, driven by things such as digital cameras and personal mobile devices, and coupled with social media connections which now seem second nature, it became apparent that the overhead needed for file-based storage of media put huge bottlenecks in the processes of moving from one medium’s format to another. Another method for storage was needed—that methodology is called “object-based storage.”

Fig. 1: Advantages in utilizing object storage for unstructured data in archiving, cloud and geo-dispersed applications.

ALTERNATIVE STORAGE
Object-based storage, which is specifically formulated to address unstructured data, is the new alternative to file storage. This relatively recent “object-technology” is built around the concept of extended metadata and collected sets of associated data. An object can be thought of as a container—a “wrapper-like” entity that captures the unstructured data and its applicable bits-about-the-bits (aka “metadata”)—and houses it in a storage space that is designed for easier retrieval than file-based storage (Fig. 1).

In object storage, each object will be assigned a unique identifier that enables servers to retrieve it from any physical location. This is a core principle in object storage which is how cloud-based data is managed on a global basis.

Object storage systems—sometimes called “object stores”—can be software-only or hardware based. Smaller stores are typically hardware-based, but large data center size solutions will employ software-based solutions that allow the storage to be deployed on a much broader basis.

SCALABILITY
Object stores provide infinite levels of scalability, something that traditional file- or block-based storage systems cannot equal. Still there are challenges in making object stores work in a block- and file-based domain.

Interfaces continue to be a key component in making file- and block-based external shared storage successful. The two external shared storage system protocols (block and file) have flourished primarily because they are as widely used and are as available as the networking interfaces that drive them.

Traditionally, block-based storage solutions have utilized Fibre Channel and Ethernet (iSCSI) as their interfaces. For file-based solutions, the interface is usually Ethernet (e.g., CIFS/SMB and NFS).

Due to issues surrounding data protection, indexing, and addressing in large-scale data repositories, block and file (and RAID) are not very well suited for data center size storage applications. Disadvantages include RAID’s inability to scale sufficiently (and efficiently) and file-based protocols that run into issues with metadata management when the storage volumes approach petabyte-sized data and/or contain multiple billions of files. We add, however, that some enterprise-class storage providers have developed high-performance provisioning for billion-plus files or when capacity reaches or exceeds five or more petabytes.

Despite its advantages as an answer to storing data at the multipetabyte level, user applications still expect to see the functionality of the more traditional NAS or SAN interfaces. These needs complicate object store integration, making them less than straightforward compared with using block- and file-based systems. Nonetheless, object stores have taken off in popularity, especially in the last 2–3 years. Object store providers offer a variety of options for making this relatively new storage form work with key applications.

DISPERSED STORES
Object stores offer a means to provide dispersed (locally in the data center) and geo-dispersed (across countries and continents) data protection. Object stores do this without RAID, using protection mechanisms typically employing a form of erasure coding—otherwise known as forward error correction (FEC).

Lost or corrupted data can be recovered using a subset of the original content which, through algorithms, let the system reconstruct those lost storage elements mathematically; often in the background and with minimal disruption.

Erasure coding is far more scalable than RAID and is more efficient (time wise) although at a cost of additional CPU overhead. From a business continuity/disaster recovery (BC/DR) perspective, users benefit from erasure coding by allowing these “subsets” of erasure-coded data to be distributed geographically in distant locations or on adjacent floors (or buildings) on a campus-wide system. Object stores also offer failure protection capabilities, another feature leveraged when installations have more than one location for their storage platforms. Failure protection (as background rebuild task) is an erasure-coding feature void of the complexities, significant down time delays, and risks encumbered when having to rebuild RAID arrays employing multiterabyte hard disk drives.

Storage systems always run the risk of data loss or corruption. Very large-scale data repositories face this problem just like smaller systems. Most spinning disk and solid-state storage media are reliable, but not totally error-free. When storage media does fail, it may be because of silent (unknown) corruption or issues that result from unrecoverable read errors (URE). This obviously places all the data at risk.

SCRUBBING THE DATA
A technique called data scrubbing helps to validate and then rebuild potentially corrupt or missing data. Erasure coding algorithms along with the typical “write-once” (read many) nature of object store data enables failed data to be recreated in the background, with little or no impact to operations. This is another reason why object stores are becoming the first choice for long term, deep archives—as well as near term cyclical storage.

In a future article, we’ll take the next steps in discussing configuration, performance balances, and the chief advantages to an object store’s rich metadata management.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

No organization or individual is immune to the problems of limited storage capacity. However, many are finding new means to adapt their storage resources to alleviate the impact of not having sufficient storage to meet their operational and business needs. Some have moved to cloud storage for their archives or as mid-tier storage silos or for temporary “bursty” needs. Others have looked to virtualization technologies to help support cross utilization of both storage and compute resources.

Virtualization is bringing to industry new methods for the allocation of services across pools of resources ranging from servers (for compute and processing) to high-performance storage (for live work) or near-line (or deep) archive. Storage resource management processes help users make appropriate decisions. Applications vary from real-time storage environments to short-term “slow” storage to long-term “deep” storage for seldom-accessed archive purposes.

When organizations choose not to use public cloud services for storage or compute resources, they have likely either built their own “private” cloud or have already carved out a portion of their datacenter for on-prem storage. Storage management and virtualization can be applied to your on-prem resources in the same way as cloud solutions, but usually on a smaller scale.

DISSIMILAR STORAGE
No organization purchases “all the storage they’d ever need” at one time. The probability that there are differing sets of storage systems in any one datacenter or equipment room is high. In these cases, the groups of storage were probably acquired at different times, for different needs, and are most assuredly of different capacities, performance and drive denominations or configurations.

Fig. 1: Conceptualization of a storage system pool with surrounding components. The “pool” consists of various sets of existing disk arrays, which are combined into a single storage pool and managed through the virtual systems director.

Varying assortments of storage (see Fig. 1, yellow box) create sharing problems that aren’t easily managed by administrators.

In broadcast media and entertainment, post-production editing systems continue to scale in performance and demand. Recently storage and server vendors have made a renewed push toward uncompressed editing and to employ UHDTV/4K shooting and posting—driving storage capacities much higher than ever before. Any of these activities infer that storage systems will need continued updates with newer, faster arrays, and more capacity to support these higher resolutions.

The quandary of what to do with legacy storage systems—besides abandon it—is a question that has many factors to consider. Are the systems so old they can no longer be supported? Can they migrate older storage systems to other purposes when adding new, higher-performance storage systems to address new workflows? Is there sufficient life remaining on the existing storage system to retain value as a “pooled” resource?

Answers vary and need new considerations as storage management solutions make retention of existing or current storage systems more viable.

As capabilities in virtualization get easier to deploy, manage or become more cost-effective to implement, it makes sense to review opportunities for pooling of individual storage systems into consolidated sets of storage resources.

POOLS AND POLICIES
A storage system pool enables the grouping of similar storage subsystems as a managed, accessible set of storage, i.e., the “storage pool.” Using virtual systems directors (Fig. 1, green oval), users create a storage system pool of selected, available and appropriate storage subsystems. The virtual director, essentially a resource manager, lets authorized and authenticated users add storage to a storage system pool, permanently delete a storage system pool or edit a storage system “pool policy.” The storage system pool policy determines how the storage volumes within the storage system pool are allocated. Allocations are based on RAID level, bandwidth and storage pool preference settings.

Basic storage system pool management begins with discovering the available storage systems and resources—sometimes called the “audit.” Before beginning any automated (e.g., audit) process, users and administrators should be sure they’ve purged any latent files from the systems. MAM systems likely have garbage collection procedures built-in, which clean up duplicate, expired or scratch files. Conversely, simpler desktop editing systems usually won’t have this sophistication.

SCREEN AND ALLOCATE
Administrative procedures, whether manual or automated, should screen all the storage subsystems for any of the hundreds of file types, which may be written to servers or libraries—such as executables (.exe), MP3, GIFs, gaming or other files in quarantine. Once purged, the storage system pool process inventories of all the storage systems, reviews the applicable licenses (e.g., virtual machine control licenses installed or the time left in a complementary evaluation period, if applicable). If needed, the inventory process may allow users to purchase licenses for additional needed storage and may check for updates related to the management pool application.

Next comes the allocation process, which, once systems are appropriately configured, may be the manual segmentation of storage groups; or with a software-controlled environment, may be a “real-time” storage management solution provided by a vendor or a virtualization solution.

TEMPLATED OR REAL TIME
Storage in a virtual environment is usually managed in one of two ways: through the use of templates (or wizards), which will be “activated” in the future; and (b) in real time—where changes happen immediately or during the next restart.

When configuring storage using templates, you only create storage “definitions”—otherwise referred to as “configuration settings.” When creating “storage device definitions” as templates, no storage device is actually configured or altered. Before the definitions can take effect, they must first be “deployed.”

How the configurations are deployed depends upon how systems administrators have configured their deployment activities. By delaying deployments until qualified, any unintended activities, storage misallocations or those without authorization are prevented.

Conversely, when configuring in real time, the storage device will be active and connectivity would already have been established. All included devices would have to be discovered and any device the storage attaches to cannot be locked or made unavailable. When changes to the storage device are ready, the user opens a “panel,” which will make those changes immediately upon clicking “deploy” or “apply.”

AUTOMATED ONLINE STORAGE
When your internal/on-prem storage allocation reaches capacity, then alternatives will be needed. One alternative is “automated online storage.”

Traditionally, the managing of online storage was by “time interval scanning” of the overall workspace. Next the system performed automated checks to see how much storage remained, the amount of storage consumed during the last interval check, and included additional safety checks for unexpected situations.

Today, statistical data derived from such activities is pushed to integral databases and trending applications that watch current and past activities, then predict what may be coming. APIs may leverage third-party products, i.e., schedulers or calendars, to help analyze upcoming needs and predict future storage requirements.

With today’s unpredictable demands placed on storage, we must ensure there is sufficient storage to handle unexpected events or anomalies, which could occur between scans. Should a need for storage exceed the available allocated storage volume, resource managers will quickly add the needed storage—plus any overflow buffer—in real time. Calls to outside resource templates that stand “ready for deployment” may be enabled automatically, either on a local basis or through remote activation into the cloud.

Real-time “management implementation” may help reduce costs by mitigating the repeated environment scans and by providing up-to-date utilization analysis of data, which is then applied to maximize storage utilization rates.

Making the most of storage resources is now a routine part of overall systems management. Effective storage management requires that the correct technologies be blended with information technology practices applicable to the business’ activities (e.g., content collection and ingest, MAM, editorial and long-term productions or playout).

With the right tools, IT managers can then optimize the storage; plan steps to safeguard the data; and in turn, help reduce operational and long-term capital expenditures.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Storage systems, whether or not coupled with editing systems, MAMs or other production-related data systems within broad- cast facilities, are all headed in the direction of IP-based infrastructures.

Regardless of whether users are leveraging fibre channel or Gigabit storage, cloud technologies or simply augmenting their conventional on-premises systems in the central equipment room—the technologies now dominating the next-generation approaches to facility integration are becoming IP-based in every aspect.

Technologies for the use of IP in real-time media networks are rapidly moving into the production and routing infrastructure domains as we steadily progress towards implementation of the developing standards, including SMPTE ST 2110 and ST 2022-6.

WHAT’S LACKING?
What may generally be lacking in the “grand plan” for broadcast facilities is an ideal, consistent and interoperable network management system that can provide not only the control structures for studio video over IP (SVIP), but could become the logical extension to services like storage management, business process/workflow, direct uplink into the cloud and many other potentials.

There is still a lot to be accomplished in this area. Through the outstanding efforts and continuous work by industry forums and associations, including the Advanced Media Workflow Association (AMWA) and the Video Services Forum (VSF); alongside the Alliance for IP Media Solutions (AIMS) and in conjunction with the Joint Task Force on Networked Media (JT-NM), the industry is helping to shape the requirements and recommendations that can make the difference in flexible, reliable interoperability on a network-centric IP architecture.

Tasks essential to interoperability and sustainability (i.e., the ongoing management of systems at a software-defined level) are generally grouped under the heading of “network management.” IT professionals understand the basics of the processes and unfortunately often approach their needs and objectives through the grueling process of trial and error.

For many well-designed network centers, strict adherence to network management is what keeps the systems functioning. In systems where administrators have not been fortunate to have an overarching plan due to budgets, staff changes, inexperience or such, network management can be perplexing and is likely approached on an “as needed” adventure through the day-to-day operations of the entity.

In the upcoming paradigm shift into SVIP on Professional Media Networks (PMN), the trial-and-error approach will not work. As a primer to what network management involves, we’ll now look at the fundamentals of network management, sometimes referred to as network management systems (NMS), in IT vernacular.

Fig. 1: Processes and procedures typically employed in a network monitoring system (NMS), which are applicable in real-time studio video over IP applications and for storage management systems.

Network management consists of these factors and derivatives: configuration, fault, security, and performance, and accounting management (Fig. 1). The depth and practices involved in these network management tasks can be applied in differing ways depending upon the size or scale of the network involved, and the types of traffic that flow on the network.

CONFIGURATION AND FAULT MANAGEMENT
Knowing the impacts of varying versions (including updates) to software and hardware is a key objective in configuration and fault tolerant management. The processes involve monitoring the network and system configuration information so that the effects on operations within the network can be tracked and managed accordingly and consistently.

Important to these objectives is configuration file management—the verification that new config-files do not degrade the integrity of the network before an actual implementation takes place. Inventory and software management includes the discovery of all the network devices (a dynamic process that lists devices found on the network) and in some cases, an analysis of the software versions present (and past) so that regression testing, if necessary, can be administered without undo impacts on the network or its associated/connected peripherals.

Fig. 2: Steps in fault management and resolution. In Professional Media Networks (PMN), these steps may be provided on an audit network, which is specifically configured as a “watchdog” and “reporting” system that is designed to detect anomalies that humans and visual perceptions might not see or find.

In the PMN, fault detection (see Fig. 2 for fault management steps) may be captured directly by devices and reported to the controlling system. Trend analysis and awareness, along with integral audit networks (described later), can aid in the processes of fault detection, prevention or correction, especially in media-centric IP-video networks.

PERFORMANCE MANAGEMENT
Two areas that impact performance management are: (a) the monitoring, measuring and assessing, plus the reporting of those metrics established pursuant to the network design and functionality; and (b) the service level agreement in place when third-party providers are involved.

Storage management may use applications such as a MAM as the “orchestrator.” The process looks at how well activities such as file-transfers between storage tiers are working; migration to archive; backup systems; or delivery of files to the workstations.

The tasks are continually monitored in the background and are quite useful to the system administrator. As workflows and work processes change (an often-regular activity in content creation and delivery applications), the level of activities required by certain sections of the MAM and storage systems may be improved by adjusting parameters, workflow steps or the time of the tasks, accordingly. This is where performance management is valuable and functionality is often available in enterprise-level MAMs and storage platforms.

When services rely on third-party resources, such as network connections (WANs or MANs or when a service provider is engaged for other services, a written agreement between the provider and the customer—an SLA)—will contractually bind the expected performance level of network services. SLAs consist of agreed-upon metrics that should be realistic and measurable for each side of the contracted parties.

At a device level, SLA performance metrics may include CPU utilization, big buffer/medium buffer allocation, misses or hit ratios and memory allocation. Device-level performance statistics are critical to gauging and optimizing the performance of protocols that drive applications and computation power at higher levels.

For network routers, switches, aggregation devices and other components that support these various higher-layer protocols, performance statistics and technologies for intercity and intrafacility functionality should be monitored, collected and reported to gauge the effectiveness and efficiency of the network and then referenced to the SLA contract.

TUNING AND ANALYSIS
Compared to conventional IT or back-office data traffic, the flows on PMNs will increase significantly and be quite “dense”—that is, signals will be continuously running at or near the interface data link rate (e.g., 10 Gbps or 25 Gbps). Backbones connecting the aggregation points will run at rates in 40 Gbps and 100 Gbps profiles. Such activities will place a high demand on network resources.

For routine workflows and applications, the rates are unlikely to waver—that is, they will be consistent with the uncompressed video data rates for the applicable resolutions (i.e., multiples of 3 Gbps per 1920x1080p stream or dual-stream UHDTV/4K at 12 Gbps per path). PMNs, as a natural course, will be tuned (i.e., configured) for specific protocols and for prescribed data rates, buffer levels and essence flows. They will not be expecting variations in the streams unless specifically requested (re-subscribed) by the control systems.

Historically, network managers typically had only a limited view of the types of traffic running on their networks. In PMNs, where real-time services cannot be burdened by outside non-media-related influences—the network traffic will be confined to specific forms and structures.

Back-office traffic (email, office applications or social media) must be strictly forbidden on PMNs; in similar fashion to enterprise-class storage systems (using fibre channel or Gigabit Ethernet) where dedicated devices are optimized only for the storage data traffic specific to media files. Control and metadata for storage solutions are carried on a separate network, through separate switches and never cross into the storage data traffic itself.

Traffic profiling technologies provide detailed views of the flows in the network. In the IT world, two familiar technologies (RMON probes and NetFlow) enable the collection of these traffic profiles. In the PMN, this process will likely be vendor-specific and tailored to the exact forms of essence (video, audio and ancillary data) that the network will carry.

Accordingly, “big data” analysis techniques may well find their way into the next-generation PMN, providing new analysis insight into the real-time topologies.

SECURITY MANAGEMENT
Controlling access to network resources, according to provisioning guidelines established by the users or administrators, is the target goal in security management. Security involves regulation and monitoring of authorizations and authentications, and provides accounting (collecting and reporting) for billing, auditing or other designated purposes.

The PMN, like conventional data networks, cannot be subjected to the risks of intentional or unintentional sabotage. Processes in the security management subsystem monitor the normal routine operations (e.g., users logging onto a network resource, refusing access to those who enter inappropriate access codes).

PMNs will have to deal with influences such as USB files being added to a graphic system that then permeate through the PMN with unintended consequences, as well as unauthorized changes to the network topologies (e.g., the addition of an IP-camera, which isn’t automatically registered into the system).

Audit networks, intended to record and track activities, changes and even collect unrecognizable anomalies (e.g., minor network glitches, dropped packets, temporary and short-term oversubscriptions or excessive latency) may become a mainstream part of the PMN going forward.

Numerous other concerns and issues will become obvious as IP crosses the boundaries from file-based workflows into real-time dedicated (PMN) solutions. Users will soon need to develop sets of “best practices” and model configurations (topics beyond the level of this article at this time).

Needless to say, for PMN applications, users should leverage available manufacturer resources and/or consult with experts (e.g., systems integrators) who are experienced and familiar with all the aspects of security, control and IP implementation.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Latency is a continual concern, impacting nearly all forms of media and communication systems. It can be evident in an internet connection, a server, a disk system and in an IP network. We’re hearing a lot more about latency as we move from the SDI-world to real-time video over IP.

Latency is defined as the time interval (or period) between a stimulus and a reaction or response. Latency can be a “physical” property—as in the velocity of propagation (VoP) in a cable referenced to the frequency of the signal carried and is based upon the materials used in the fabrication of that cable.

Latency can also be a “signal transport” property, such as in the time of flight (ToF)—i.e., the methodology describing the time it takes an object, particle or acoustic, electromagnetic or other wave to travel a distance through a medium.

Fig. 1: Typical causes of latency due to propagation delay of the links (Dlink) and in processing delay at the nodes (Dnode) which must be accounted for in system design. ToF might include how the elements in a network (transmitters, receivers, switches and transducers) are engineered; while VoP is influenced, in part, by choice of cabling itself (e.g., Belden 1694A compared to 1855 for video or Cat5 vs Cat6a for IP).

Video latency—especially in a packetized IP-network—can affect reliability, stability and the ability to properly switch a line of video. Practices described in SMPTE RP168, the recommended practice defining the vertical interval switching point for video, help govern how real-time video-over-IP systems address the packetization of essence for live (real-time) video systems.

The needs for a live stream of video compared to feeding a monitor wall or encoder take on different perspectives, which must be carefully considered as signals negotiate segments of any network.

Latency in the recovery and reassembling of packets in a video signal transport network can affect the time it takes to reconstruct a complete frame (field) of video. Too much latency and you have a time inconsistency causing skipped frames or stuttering video.

Buffer capabilities also affect performance. If the network switch’s buffer is insufficient, the impact can result in an incomplete packet resolution, which could cause a series of video frames to be lost or a switch-routing or connection to fail—thus generating another improperly displayed image or a total loss of signal.

Performance and latency go together when looking at any system solution or application, which depends upon consistent and deterministic delivery of data. In today’s media implementations, audio and video essence (the data, sometimes called the payload) can take on variances depending upon where the data is presented in the workflow. One relevant and timely perspective (for the professional video industry) is embedded in the Real-time Transport Protocol (RTP) characteristics of networking; i.e., how the signals are impacted in a streaming-video vs. a file-based workflow.

Interested technical readers should read and understand such terminologies as RFC 3550 (the transport protocol for real-time applications) and RFC 4175 (the RTP payload format for uncompressed video)—both openly available and developed through the IETF (Internet Engineering Task Force).

SERVER PERFORMANCE

Servers bring with them their own set of latency and performance issues. Insufficient memory sizing; poorly managed caches or processor core(s) that cannot keep up with the signal throughput demands, which will result in a slowing of applications; signal throughput delays; or, in the worst case, computational errors, which hinder synchronization with the overall system’s demands.

Multicore servers available today have plenty of CPU power; yet getting that power to or from the network may be an issue. Network interface cards (NIC) and host bus adaptors (HBA) can be a server bottleneck if not properly configured, specified or implemented.

Traditionally these I/O components were locked to a single core processor. However, by using other resources, such as hypervisors and resource-side scaling (RSS), performance is increased by allowing the interface cards to distribute the I/O processing across multiple processor cores.

Another server virtualization technique deals with sorting the I/O tasks to the right virtual machine. This technique involves what is referred to as “virtual machine device queuing.” This VMDQ technology is productized by vendors who make I/O and processor cards. It allows the Ethernet adaptor to communicate with established hypervisor products to group packets per the best needs of the VM they are intended to be directed to.

STORAGE SYSTEMS

Storage solutions intended for media production have their own set of throughput, performance and latency issues, which are generated by varying factors. At the single disk root-level, and the most notable or easy to understand, are the impacts of “rotational latency” (sometimes called “rotational delay”), which is based in part on the rotational speed (RPM) of the drive.

This is a physical factor prevalent in every spinning magnetic disk or optical disc drive medium. Rotational latency is measured as the time that it takes the data on the drive platter in a sector to be positioned beneath the head for the read (or the write) process. The worst-case time-period number is found when the drive data has just passed the head, and the command to retrieve that data can only be met when the drive needs to complete a full revolution before being properly positioned to read that data.

Other factors include “seek time,” the time for the actuator arm to travel to the proper track on the drive; and “access time,” also called “response time,” the time it takes for a drive to transfer the data from the specified track to the drive electronics.

These factors are cumulative and affect performance, but in recent times we’ve seen additives to the storage mix that help alleviate performance-latency issues.

Solid-state devices, as complete replacements for HDDs or as supplemental caches for HDDs, help manage performance issues by (in the latter case) putting frequently accessed data into SSD caches. Here, routine calls for operational data or system metadata are stored in SSD devices, so accessing the HDD is reduced, improving efficiencies.

When editing applications know that they will frequently be playing a clip from the primary timeline, the well-designed app often instructs storage management to put that data into SSD. Timeline performance is improved and latency is decreased because the SSD has a significantly faster access time than does the HDD.

EVALUATING AND MATCHING

Storage systems should be demonstrated per the user’s specific implementations. For example, a NAS system designed specifically for media and entertainment will employ characteristics that can’t be met by other non-M&E storage solutions. One known evaluation test used in storage demos is to compare the load time of a set of files from a central storage array to the active NAS used in editorial or shared collaborative production.

Another is to check the ability to accurately scrub the video and audio in a high-resolution file (e.g., a 4K full-resolution sequence) and see how closely and smoothly the audio tracks the video.

Significant differences can be achieved by marrying the appropriate operating system (OS) to the file system, which traditionally have separate roles. When the file system is aware of the underlying disk structure, values are added.

In the past, file systems were traditionally created only on a single disk at a time. When two (non-RAIDed) disks were employed, two separate file systems would be created.

In a RAID configuration, this situation is avoided because the OS looks at only a single “logical disk,” which is likely comprised of many physical disks, on top of which the operating system places a file system.

In software RAID, the file system living on top of the RAID transform sees it is dealing with only a single drive. The implementation reduces read/write process complexities and hence improves performance. One pertinent example is ZFS (originally developed by Sun Microsystems, now Oracle), which is conceived of a combination volume-manager and file system. The concept allows the creation of many file systems that share a common pool of available storage. The storage system could be grown automatically because the file system is aware of the physical disk layout. This permits existing file systems to be grown automatically when additional disks are added to the pool. The new volume-space is then made available to every one of the file systems.

STORAGE IMPROVEMENTS

The evolution of storage “additives” is what storage vendors and solutions providers continually do to “build a better mouse trap,” as the colloquial saying goes. Users will inevitably keep changing their storage solutions because the improvements make consequential differences in their workflows. Pooling resources and using a storage manager product to relegate where that storage is used in the enterprise is just another way to extend the usefulness of legacy and in service storage.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Since storage has become a key component in nearly all media systems and workflows, one might wonder if “storage analytics”—the detailed information about individual and overall storage elements—are important to operations and management of the entire media production ecosystem.

Utilizing and understanding storage analytics is becoming more important as users configure virtual environments (such as VMware or Hyper-V) as well as conventional structured storage systems. When virtualized computer environments take shape across the enterprise, shared storage can also take on a much broader dimension. While multiple operating systems can operate from a single physical computer, how the information (data) generated in those environments is stored or managed becomes more difficult. Complex multi-element environments are never as easy to understand or manage compared to how a single OS, computer or server platform addresses a small NAS or group of drives.

Virtual machine environments and virtualization fabrics are subjects which have much depth, vary by manufacturer, and—for now—are beyond the scope of this month’s article. However, whether your system is VM-based or a mid-to-modest scale NAS or SAN, the value proposition in having analytical tool sets available that can manage the overall storage system is becoming a subject of interest to many IT managers and production workflow administrators.

If you’re fortunate enough to have a greenfield system with a full complement of new “raw” storage, then the options to configure and deploy this untapped storage pool are many. Yet, if your storage pool is a collection of many different sets of storage (as with many facilities) that are accumulated over various time periods, the efficient management of those storage “islands” takes on another dimension altogether. Often such systems may be comprised of DAS, NAS or SAN elements and will have little consistency in terms of volume sizes, I/O capabilities and distribution.

PERFECT CANDIDATE

The later (likely more typical) configuration is a perfect candidate for both storage virtualization and for a toolset capable of looking at the demands and stresses put onto the storage system. This process that uses these toolsets, in more modern terms, is known as “storage analytics.”

With the popularity continuing to grow in cloud technologies, a quick Google search for storage analytics will yield multiple sets of products provided by major cloud providers and by enterprise class storage vendors. Most products or services are prefaced by the cloud provider’s name (e.g., Azure) followed by the marketing term for the analytical toolset.

Some of the storage vendors will wrap the sentence to include VMware, Big Data, all Flash, scalable or some other storage marketing buzz word. The plethora of terms sometimes breeds confusion as to what you’re getting and how well it really does its job.

So, what’s under the hood? Why are storage analytics important? And what do they do or provide? How do they benefit the user?

STORAGE SYSTEM INSIGHT

Storage analytics are intended to provide insight into the physical and/or the virtual storage environment. Its tools should allow the user/administrator to optimize the storage system regardless of the complexity or the distribution. Additionally, there is also a business-sense component defined as “improving business performance” and “reducing costs.”

Fig. 1

A storage analysis tool set is designed to complement (i.e., improve upon) the traditional storage vendor-provided management systems. Analytic tools can help discover unclaimed usable storage across all the LUNs or volumes in a storage system, potentially providing significant cost-savings to any scale organization. When unclaimed storage is repurposed for other functions, users in turn may avoid having to increase storage capacity or might actually find performance is increased based upon what that orphaned storage is repurposed to do.

Fig. 1 illustrates an end-to-end data/storage analytics system, which consists of multiple probes within the storage and virtualization hardware (diagram’s left side) which collect the data via an FTP (or similar) server. The “Analytics Processor” core and database system (diagram’s right side) is the software-based solution that schedules and generates user reports based upon sampled information collected from the probed devices. The various processors in the core then create the trending, alarming and other data—reporting it to the Users’ fixed or mobile devices via web-based interfaces (concepts courtesy of Hitachi Data Systems’ data center monitoring solution). An optional BIRT interface (business intelligence report tool) may also be added, if desired.

VELOCITY, VARIETY AND CHURN

Trending analysis is another important tool found in most storage analytic toolsets. Key factors that may be helpful include the capability to look at data churn; that is, the “frequency” of change or the “retention time” of the data—information that can be used to adjust hierarchical storage management (HSM) policies or which application uses which data pool sector. As for the “velocity” of the data, is the system I/O optimized for the storage solution? Are there bottlenecks that hinder data transfers from one LUN (array or volume)?

Also in the toolset is a view into the “variety” of data—i.e., how much of the data is metadata, how much is full-resolution images (video, for example) and how much is proxy resolution (stills, short clips, thumbnails). Since storage should often be optimized for the type of data (small chunks, modest blocks or gigantic contiguous streams of files such as for 4K images), it is helpful to understand where those kinds of data are being placed, especially when you’re merging smaller volumes onto a larger storage pool.

TRENDING AND PERFORMANCE

Performance statistics and reporting are vital components in any suite of analytic tools. Many storage vendor tools provide basic, ad hoc reporting on storage performance. Few provide detailed or in-depth analysis with different levels of user-configurable granularity. The amount of detail available (i.e., the granularity of the information) should be adjustable based upon the system needs or the purpose of the investigation. The statistics used to generate the reports will then meld into the trending analysis programs, so the term of the trend and performance reporting needs to span multiple years.

At some point, storage will need to be either replaced or upgraded (expanded). Trending tools help predict where that point is so that the organization is not caught having to invest in more storage at an inappropriate time. The data points may also help determine when a cloud storage application is appropriate and to what volume (capacity) is proper.

BUSINESS INTELLIGENCEGATHERING

Ancillary products—those generally found useful in business intelligence reporting—are now being adopted throughout data centers and enterprise-wide large storage systems. One example, called Business Intelligence and Reporting Tools (BIRT), was developed as an open source software project aimed at providing reporting and intelligence via web applications created for rich client platforms (RCP). It integrates well with existing Java-based reporting applications. BIRT can be found integrated into some vendors’ storage analytical tool sets and by those who provide data center class storage as well as media-focused storage solutions. For storage analysis applications, BIRT is especially valuable if the organization is already employing BIRT methodologies throughout its other business units.

Intelligent analysis of your storage solution sets is valuable at any stage of the storage life cycle. At configuration, the tools help you analyze and optimize actual performance. At mid-term of the life cycle, the tools will provide benchmarks for peak performance, which can be used to validate or justify the storage solutions’ continued use or tweak its applications for improved optimization (should workflows change or be added).

Throughout the many storage work cycles, these tools help deliver consistent and concurrent reporting on the storage system’s status. And finally, the tools help to solve the most difficult of performance issues in a timely and convenient manner.

When selecting a storage solution for your organization, ask the vendor about the storage analytics tools available from the provider; then ask which third-party solutions they recommend or know have been added to their systems by other end users. The use of these storage analytic tools, from the start, may make a difference to your storage life cycle management two or more years down the road, and can be used to leverage change or improvements enterprise-wide.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage topics, read his book “Moving Media Storage Technologies.” Contact Karl at kpaulsen@diversifiedus.com.

Over the course of the past decade we’ve seen a proliferation of servers, often as commodity-based products from the usual sources, being added to facilities for nearly every system in the video production equipment room or the delivery network.

Servers—in a general sense—have steadily become the defacto device that provides the operational engine to much of the functionality seen in the modern digital media age.

From an implementation perspective, design practices have layered consecutive sets of hardware (e.g., servers and storage) to the mix of already a dozen to upwards of hundreds of other servers, each with a purpose aimed to support the next set of software applications or implementations.

Historically, each server would impart its own dedicated purpose within the workflow. Sometimes you would find the same functionality across different servers; see them in augmented segments of ingest, post production or news; and now we’re seeing them utilized extensively throughout the entire delivery chain.

With each device comes a primary connection and configuration to a network. Sometimes, depending upon the facility’s directives or budget, another secondary network interface to another network section would be added for protection or overall system resiliency. This continued duplicity and individuality of servers and fixed network topologies is costly and cumbersome to manage.

Fig. 1: Software-defined network (SDN) reference architecturePROFESSIONAL MEDIA NETWORKS

Virtualization is changing this legacy model. When similar functions can be distributed amongst a pool of servers whose functions can be modified to serve other purposes once a work task is completed, the efficiency of the system improves and the CapEx and OpEx costs are reduced. Virtualization, among other factors, is aimed at reducing the hardware components in a facility without reducing performance or other workflow support. Yet there is another evolving concept that has grown out of the compute (IT) industry and is filtering its way into professional media networks (PMN) for file-based and real-time video over IP.

This month’s introduction to SDN follows up from my April 2016 column on data- defined storage. In the IT industry, where entirely software-based systems are rapidly becoming the norm, the organizations that wish to accelerate application deployment and delivery—while dramatically reducing their overall IT costs—are transitioning to what is called “software defined networking” or SDN. Essentially, an SDN is policy-enabled workflows built on the principles of automation. In cloud-based systems, SDN technology is the catalyst that enables the cloud architectures, which provide for automated, on-demand application delivery, portability and mobility at scale.

DATA CENTERS AS CLOUDS

In a data center, the economies of scale are achieved in part through virtualization—that is, the sharing of like component to provide services as requested by the enabling software- based control and management systems. Referred to as “data center virtualization,” this functionality leverages SDN to provide for increased resource utilization and flexibility while in turn reducing the total costs for operations, including infrastructure costs and overhead.

Data centers can be considered clouds that may be private (i.e., services provided for internal organizations); public (i.e., any-services [XaaS] provided to others for a fee); or hybrid (where excess capabilities built for private services are sold publicly to entities outside of the internal organization). Regardless of the scale of the cloud data center, they all function primarily the same.

Typical cloud computing infrastructures— large sets of identical or very similar servers—live on large blobs (Binary Large OBjects), which form the common server structures. The blob is a collection of binary data stored as a single entity in a database management system. Blobs are usually composed of images, audio or other multimedia objects. Binary executable code may also be stored as a blob.

In a data center/cloud environment, myriad servers evolve much faster than in a traditional IT infrastructure built for specific functionality (such as for compute, back office or the like). Thus, blobs are not homologues (i.e., they may not be corresponding or similar in position, value, structure or function) to one another. In this scenario, server groups are fundamentally identical. For example, they are x64 architecture-based and they all share similar features, such as the peripheral component interconnect express (PCIe), serial advanced technology attachment (SATA) and Ethernet interconnects, which are essentially structured roughly the same way over repetitive subsystems.

This gives the data center/cloud the capability to scale—the capability to grow quickly without having to continually modify the physical (or software) infrastructures to “scale-up” to meet demands.

The management component for the orchestration of these capabilities is built on software-defined solutions including network components and virtual machines.

WORKFLOW DEFINED

An SDN allows systems to develop and adapt to varying workflows—that series of activities that are necessary to complete a task. Workflows are usually comprised of a series of stages or steps, each with a distinctive step before it (except for the first step), and are followed by another step after it.

Workflow steps are usually linear, but may also include looped steps with decision trees that allow the sequence to exit the loop and continue to the next step (or workflow) upon meeting certain conditions.

Workflows can be complicated and should be documented so they can be understood by external (human) sources or meshed with computer software built on a human-readable coding structure such as XML (eXtensible Markup Language). The written documentation is based on business processes modeled using principles found in business process management (BPM).

HYPER-CONVERGENCE AND ORCHESTRATION PLATFORMS

For a cloud or cloud-like implementation, the programming is called the “cloud orchestrator.” This platform manages the interconnections and interactions among cloud-based and on-premises business units. Orchestration may be applied to one or many sets of servers, but its best use is when applied to a system of servers, which, by themselves, have almost no understanding of what the adjacent server is to accomplish now or in the future.

Business objectives (e.g., a global ingest platform that brings content into a central repository) drive how the orchestration works and how workflows are developed (Fig. 1). SDN accomplishes these objectives by converging the management of network and application services into centralized, extensible orchestration platforms that automate provisioning and configuration of the complete infrastructure. Centralized policies collectively bring together disparate groups and workflows so they can deliver new applications and services in minutes, rather than the days or weeks required in legacy systems.

Depending upon its scale, this concept may also be known by the term “hyper-convergence”— a form of infrastructure with a software-centric architecture that tightly integrates storage, networking and virtualization resources (alongside other technologies) in a commodity hardware-based system; usually managed under some form of SDN.

SPEED, AGILITY AND FLEXIBILITY

When deploying new applications and business services, SDN enables the system to deliver speed and agility using existing components consisting of servers, storage and network switching. Programmability of these components is a key characteristic of a software-defining solution.

Another way to look at this is when various applications reside on servers that can be administered (delegated) when or as needed by the orchestration platform—while the associated server communications, instructions, process loading and data-steering throughout the network is being managed by the SDN.

SDN concepts are core components to making the emerging studio video over IP (SVIP) systems work using common Ethernet switches that can be configured per specific standards, some of which are currently in development.

Karl Paulsen, CPBE and SMPTE Fellow, is the CTO at Diversified. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Network attached storage, or NAS, continues to expand in acceptance and in capabilities. Where, for years, the storage area network (SAN) seemed to prevail for high-end, performance-tailored storage, NAS is changing those dimensions.

Using NAS as a storage solution is practical when the content includes sources such as surveillance or smart devices. As content types and storage dimensions evolve, users are finding they need to understand not only traditional NAS, DAS or SAN solutions; but they also must be looking at some recent trends in storage that are not so wellknown.

For those who have supported conventional NAS, they may now be looking toward “scale-out NAS” as its natural alternative. And for those who may have already outgrown the efficiencies of scale-out NAS, the latest revitalization comes in the form of “object storage.”

This month we’ll look at both of the later storage technologies, touching on why they each have applications based upon factors such as file sizes, quantities, search-ability and metadata.

SCALE-OUT NAS
Personal computers typically store data that is based upon a file system; a technology that can easily store information and organize data on a structured basis and which is human readable. The concepts in scale-out NAS let this data organization operate as one large global namespace that typically keeps the data stored in a hierarchical structure.

Hierarchical storage is best explained by suggesting that the file sets are structurally located within a system consisting of subfolders that nest among parent folders and reside on a common hard disk, solid-state drive or similar storage platform.

As storage requirements grow, the scale of the file system must increase. Scale-out NAS provides a solution that allows for storage capacity and file system scalability. The NAS device is effectively like a gigantic “C: drive” with capabilities to store millions (to billions) of files.

Fig. 1: a scale-out NAS architecture integrates with the traditional, and simpler, LAN-based client/server/storage topology. Early scale-out NAS systems were built on the principle of nodes (Fig, 1). Multiple nodes can be appended to the NAS (i.e., “scaled out”) so as to improve overall storage (i.e., “scalability”); yet often at the sacrifice of small-file performance. As the node count increases, overall storage system performance begins to slow. Metadata searches can be significantly impacted as the size of the NAS and number of nodes is increased.

Flash technologies have changed the small-file performance by a technique referred to as flash-first, a metadata management scheme that improves internodal networking speeds.

Today, the ability to store billions of files without diminishing overall performance, especially for metadata searches, is the norm for scale-out NAS.

Real-time data consistency is another key factor in scale-out NAS. When dozens to thousands of users must access the NAS, the ability to lock the metadata—in real time—is critical to preventing accidental overwriting or corruption while accessing the same files.

It is not uncommon for enterprise users to access the same files for different purposes. Such functionality mandates a metadata- locking system, which allows those files to be shuffled and/or their metadata renamed depending upon what the user’s purpose might be.

Human file management is another key factor for scale-out NAS. A typical use case example of both hierarchical and human-readable metadata is in the practical structuring of photographic images, which are often renamed, shuffled or moved into other folders for organizational or identification purposes.

The original image is likely known by the file name given to it during the capture process, such as “DSCN1234.” This is a rather useless name to anything other than a database manager. Users often will copy that file and rename it, e.g., “Johns 18th Birthday Present,” placing it (and several others) into a folder called “Family Birthday Pictures,” which resides inside another folder called “Birthdays.”

Scale-out NAS allows these file-naming techniques to extend throughout the individual’s library or in the case of the human resources department in an enterprise, across many interdepartmental file shares or home directories.

Other important factors for scale-out NAS include security management (user accessibility with read/write permissions); file-sharing among varying applications across a large NAS without the risk of one application overwriting the file in use by another application; and the ability for IT departments to have a central process by which all storage is globally controlled under one management system.

However, scale-out NAS is not without its limitations. Users are finding that even modern scale-out NAS solutions cannot keep up with the accelerating demands for storage.

We’ve often believed that unstructured data was dominated by content that was video-centric in nature. That case certainly holds merit, but as the Internet of Things (IoT) becomes more prominent, machine-generated data is gradually overtaking the amount of data generated by humans. Such emerging “web-scale” growth requirements are driving changes as to how files and file systems must be managed. Object storage is one of those solutions that simplifies the manageability equation.

Fig. 2: The familiar “rigid” hierarchical folder structure vs. the easier “everything in one bucket” approach for object storage that depends on containers instead of folders.OBJECT STORAGE
As a relatively recent concept, object storage is being applied to many storage provider solution sets aimed at addressing the limitations of scale-out NAS or other storage solutions.

Object storage mitigates the more complex hierarchical metadata attributes controlled through the legacy Portable Operating System Interface (POSIX) standard (Fig. 2).

Because object storage has only a few commands (e.g., Get, Put, Delete) it is an extremely simple-to-use interface. Its simple set of commands allows objects to exist with (globally) unique identifiers that are managed in as single “flat” address space.

The underlying principle in object storage is that data can be retrieved without having to know where or how that data is stored.

Object storage is governed in part by an extended metadata set, which is much deeper than that found in the conventional file system management tool set. Objects essentially become “self-describing” (the object knows what the information is about or what it is for). Objects can contain specific details about the application it serves, without limits as to what the content is, the size of the metadata contained in the object or how that information must be interpreted.

Objects therefore contain an extremely rich set of information, which users and programmers can leverage to allow their applications (and the storage that supports those apps) to perform much better. It includes attributes that allow for global distribution and infinite scalability. Its self-healing design yields high data reliability and provides bulk storage to be obtained at a much lower price point.

Users at both the enterprise level and the individual level are finding the feature sets in both scale-out NAS and object storage can overlap each other and, for many, yield new and important functionality heretofore unachievable with a single traditional storage solution approach. Object storage devices are now beginning to take advantage of NAS-like features as well, such as the expanding of interface-access methods across multiple applications, including both file and block storage.

No doubt, this will change how people buy on-premises storage and is, in part, how the cloud organizes its enormous amount of data on a much broader, global basis. Expect to see a lot of growth in these two storage technologies in the coming years.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage and media topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Tomorrow’s broadcast equipment rooms may look more like today’s data centers . As broadcast equipment technologies move closer to information technology-based systems, the way in which we design, build, monitor and account for operational costs is expected to change in near lock step with the technology we’re going to be building—or already are. With operational costs continuing to climb, items such as power, cooling, real estate footprints and even redundancy are likely to be watched more closely than ever in the past.

Many of the considerations for cost monitoring and control are somewhat obvious. For example, the overall power usage of each equipment rack determines the amount of cooling required; but that’s not the only factor in understanding cooling and the power that it takes to drive that cooling. As these spaces begin to look more like data centers than broadcast central equipment rooms (CER); the techniques used by data centers to control factors like Power Usage Effectiveness (PUE) become sectors we will continually need to manage.

VIRTUALIZATION ATTENTION

Efficient utilization of the myriad servers now being installed in the CER can have direct impacts on balancing the power usage and practicality of each system they support. This is why “virtualization” is getting more attention now than in the past. Many of the services used in the broadcast plant are seldom turned up to full capacity. Products that were once available only in a dedicated box are now being ported to a software-only domain where they can be loaded onto on-premises, common off-the-shelf (COTS) hardware—or even placed into the cloud.

Since the main point is to get the best use out of each device, there will eventually come a point where groups of servers, which have dedicated functions—but only run 25–35 percent of the time—will become consolidated and managed by systems that allocate the services on a needs-or-demand basis.

For example, if the functions of three servers are running at peak only 35 percent of the time, it is potentially possible that you can take those services and run them on a single server by throttling the workload and managing the utilization from a virtualized, pooled resource perspective.

UTILIZATION FACTORS

Media asset management systems and news editorial systems are notorious for throwing all sorts of servers into their workforce, but really not getting much more than 50-percent utilization during average work periods. Granted there are times when certain functions require those servers to run at nearly 100 percent; but that’s when the virtualization environment makes its best showing.

Assuming there are 30 servers in a broadcast facility (and that, frankly, is now a low number) for this example, picture the server mix as a group of 10 servers doing ingest, another 10 doing processing and the last 10 are parsed between output-conforming and shuffling media to/from storage. Ingest is likely to be unpredictable, so expect those 10 servers to be dedicated 80–90 percent. The processing servers are entirely workflow-governed. Processing probably gets the lowest utilization (or “loading”) percentage, say in the neighborhood of 20–25 percent.

Finally, those servers dedicated to do output-conforming and media shuffling aren’t much better at resource utilization than processors, so they run, for example, at only around 35–50-percent loading.

It doesn’t take a lot of rocket science to see that the idle time of better than two-thirds of the servers is greater than 50 percent. So why not consolidate the services to less servers and begin to cross utilize the hardware by finding software that can distribute the loading across only 20 servers?

If the services of processing (i.e., transcoding or proxy or quality control) and those used for output-conforming could run on the same servers, then the utilization efficiency of those 10 goes from 20–50 percent up to 55–75 percent. This leaves plenty of overhead to turn up one set of services during a peak time, or turn down other services and shift the application resources to other servers that are less loaded.

SPREADING EFFICIENCIES AROUND

If you could distribute this concept across even half of the servers in a modern IT-based facility, the utilization efficiency increases and the amount of wasted power consumed by the servers when doing “virtually nothing” goes way down. In the average data center, about 50 percent of the power consumed goes to the compute side (i.e., servers doing their job). Over one third of the power goes to cooling those servers with the remaining 17 percent of the power going to other items including lighting and losses due to electrical wiring (as “IR-loss”) and such.

Fig 1: Monitored power distribution unit (PDU) or “plug strip” with serial or Ethernet monitoring, which reports the voltage, current and power factor for each outlet; and the inlet power for utilization comparison and loading per device. What can be done to monitor and control these (in)efficiencies? One solution is to install PDUs (power distribution units, aka “cabinet distribution units”), in the data center, which can monitor each power outlet and each branch circuit (Fig. 1). This data is fed to a building management system (BMS), which can then keep track of each device in the CER by time and consumption in many dimensions.

In this model, it is also important to measure the current, voltage and power factor at the inlet side of each power strip using a technique known as “per inlet power sensing.” This lets the BMS understand power in versus power used.

PEAK LOAD OR LOW USAGE

Once monitoring is configured, each device can be tracked against time and workload. Since the power draw on a server is proportional to the amount of computing that occurs, a simple power monitoring application—typically found in a BMS—can then determine when peak loads or low utilization occurs. Over time, as opportunities come to transform the “single function per device” into a “shared resource,” there may be a point where shifting certain services to different times or reallocating the functions of a pool of servers to a virtualized set of services can change the power efficiency utilization curve.

If you live in an area where part of the commercial utility bill is based upon “changes” in demand, the leveling of the power usage across the most expensive power periods could make a difference in your overall utility bill since power loading and cooling could be spread more uniformly across the day or night. Reducing the changes in the power delivered can make quite a difference over the course of a billing period.

These concepts may not seem especially easy to understand or may not make much sense today, but over the long haul and as conditions change, the practicality in adding power and building monitoring capabilities could make a difference in the operational costs of the CER or data center. Most new “greenfield” facilities are already putting the extra monitoring capabilities in place today, knowing well in advance there may be secondary advantages that will come into play years later.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage and media topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

Anyone with any type of high-performance storage system for a video playout server, play-to-air system or nonlinear editing solution of any scale has probably experienced this. One of their disk drives fails completely or you get that error warning of “imminent drive failure—change drive ooooX3Hd immediately!”

Some may procrastinate and risk certain impact. Others take heed and elect to change the failing drive, and still another group sits back knowing they’d planned ahead and bought an extra hot-spare and that the drive controller system will take over without human intervention.

Fig. 1: Characteristics of selected Redundant Array of Independent Disks (RAID) levelsRESILIENCY TO DATA LOSS
When a hard drive in RAID configuration fails there is a period whereby the system fault tolerance or resiliency to data loss decreases. Depending upon the RAID level employed or the protection scheme in place, the risks may range from moderate to serious.

The most significant and primary concern to the system is if another drive in the same array fails. Such a loss compromises the entire storage system and renders all the data in that LUN, array or possibly the system useless.

When the protective element in either the dedicated parity drive (as in RAID 3 or RAID 4) or the secondary protective parity set (as in dual-parity RAID 6) are no longer available, the period between then and when the array is rebuilt and back on line, at 100-percent service level, can be dicey at best.

When a failed or failing drive is detected, the administrator/maintenance technician must first replace the bad drive, which in turn triggers a process called the “RAID Rebuild.” RAID rebuilding is the data reconstruction process, which mathematically reconstructs all the data and its parity complement, so that full protection (with fault tolerance) is restored, in essence returning the system’s resiliency back to a “normal” state.

Sometimes when certain data checks or other errors are detected by the controller, the drive array may go into a reconfirmation period whereby checksums and/or parity algorithms perform a track-by-track, sector-by-sector, block-by-block analysis on each drive. Ultimately, parity or checksums are all compared and/or rewritten and the array is then requalified to a stable, active state.

STEPS TO RECOVERY
RAID fault tolerance involves a number of steps. In one example, should a disk fail, the RAID controller attempts to copy the resilient data to a spare drive while the failed one is replaced. Using parity data and RAID algorithms, which vary depending upon the RAID Level, parity data is then reassembled back to either the dedicated parity drive (as in RAID 3 or RAID 4) or is distributed across all the drives, as in RAID 5 or RAID 6 configurations. See Fig. 1 for selected RAID characteristics.

For other RAID levels, should one of the main data drives fail and no active hot-spare is available, a new HDD must then be installed. Then data from the other remaining drives is reconstructed using data extracted from a dedicated parity drive or from the parity blocks distributed across the array, back onto the new drive. Either way, the risks during the rebuild time are elevated until the new drive is brought online with all the reconstructed data and parity elements having been restored.

Large-scale drive arrays, those with hundreds of spindles (HDDs) usually have sufficient overhead, intelligence and processing bandwidth to compensate for certain fault issues. Those which employ intelligent RAID controllers can also be proactive. Should the RAID controller suspect or detect that a hard drive is about to fail, the proactive controller may begin the process of RAID rebuild to either a (hot) standby drive or signal the user to replace the failing drive, or add another drive in an available slot so that the RAID rebuild process can be kickstarted before an actual failure occurs.

FAILURES DURING THE REBUILD TIME
One of the drawbacks to RAID—when in failure mode or during a rebuild process—is that the performance of certain applications or processes may be impacted due to system latency. System throughput—otherwise known as bandwidth—may be reduced because: (a) not all the drives are functioning; and (b) the rebuild process takes away the I/O speed while it rapidly moves blocks of data sets from the remaining active drives onto the new/replacement drive.

The reduced performance can be especially noticeable when the array is relatively small, i.e., when the number of spindles (drives) is low; or when the individual HDDs are very large.

As hard disk storage capacities continue to increase, rebuild process times will take longer and longer. In some cases, for drives in excess of one to two terabytes, the rebuild process can last from several hours to several days. Of course, during this period, latency and the risk of another failure increases, resulting in performance and usability becoming further compromised. This is one reason, among others, that high-performance storage solutions tend to use smaller capacity HDDs (e.g., 300 GB to 750 GB) and may put many more drives into a single chassis or array.

These issues and considerations become part of the selection and decision process regarding how to choose a storage solution. For the more advanced video server or mediacentric products, those built for mission-critical operations, many manufacturers have already taken these conditions into account and provide sufficient fault tolerance or resiliency to “ride through” most of the more commonplace maintenance, upkeep and failure situations.

Another technology solution involves the use of flash memory or more appropriately, solid-state drives to either supplement the array (as a cache or secondary storage tier) or completely replace the hard disk drive altogether. I’ll explore that topic in greater depth at another time.

Karl Paulsen is a SMPTE Fellow and chief technology officer at Diversified. For more about this and other storage topics, read his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

LAS VEGAS—TV Technology asked a cross-section of NAB Show-goers a series of five questions regarding what they considered the main themes, evidence of those, whether or not these initiatives will take hold, and what promising technologies from past NAB Shows did not see daylight. (A complete list of quotes from respondents and links to their full 5Qs is at “NAB 2016 in 21 Quotes.”)

Karl Paulsen, chief technology officer of Diversified:
Q1.How many NAB Shows have you attended?
K.P. Since 1976, 39 years’ worth—maybe missed no more than two or three between 1978-1994—but continuous from 1995 onward.

Q2. What, in your opinion, were the main themes of the show this
year?
K.P. Interoperability, IP-video is here, we may be beginning to solve the automatic metadata capture issue, cloud based solutions are viable and available but the whole story is still incomplete, drones and VR are a big deal, at least for now.

Q3.What were some examples of these themes?
K.P. Evertz and Sony joining AIMS, but both holding on to some of their roots in ASPEN and NMI SAM, Lawo, and others are providing IP-centric products you can order today. Arista, Cisco and others are joining the IP-bandwagon for network switches, but for Cisco, it’s a very small part of their total product line. Audio vendors are starting to pay attention on how “their” particular adaptations of AES67 (Dante and others) may affect their next-generation product offerings.
GrayMeta and MOG are both showing means to capture metadata through entire systems and then “work on it” from a search, discovery and registration perspective. (This may cross into the IP-metadata/ANC-data domain in the next few years.)
Cloud is still everywhere. Pebble showed an actual full demo on how to spin up their systems in a full-on cloud environment (they were not alone); while others (Harmonic with VOS365 and
VOSCloud; Imagine with the same story as last year) are making headway but still have a few missing pieces.
IBM is beginning to figure into this big time; given they have more pieces of the puzzle than others, including now long-line and last-mile fiber connectivity.

Q4. Do you foresee any or all of these technologies or initiatives taking hold?
K.P. Once settled from a standards adoption perspective, TR-03 as it becomes SMPTE 2110 may drive interoperability for the next generation of video transport to success across most of the current vendors playing in this space.
How these systems are marketed and built out requires new expertise and experience, which most users/owners/facilities lack (and many systems integrators as well). How to make money from a reseller/expertise perspective is an interesting discussion. Changing the landscape for systems integrators will need to be thoroughly analyzed as we transition from a hardware/design-build to a software/consult-configure-deploy/support perspective.

Q5. What technology that impressed you most at a past show didn’t see the light of day?
K.P. Fixed-based/hardware-centric traditional broadcast automation is dying a slow-but-sure death. The quad VTR is definitely dead.