One of the lesser-realized but very important elements of artificial intelligence is real-time adaptation and decision-making. Where is this important, one might ask? The ability to process information as it arrives and then to make informed decisions without significant delay is an area where AI can be quite valuable.

Familiar applications of real-time adaptation include command-and-control environments, security situations, traffic control or monitoring environments and in autonomous driving (autopilots). Every one of these situations requires intelligent systems to be able to make adjustments in response to dynamic situations and—in most cases—in real time.

Adapt in Real Time
Any one of us can probably imagine the number of decisions that must be made in a self-driving vehicle solution. The ability for any system to “adapt in real time” is becoming essential in this fast-moving world—and AI is a primary element in those advancements.

Today, companies must be able to act quickly on data-driven insights to be more agile, proactive and to seize emerging opportunities or respond to sudden market shifts. Amazon is a good example of a business that must be able to move in a certain direction without being burdened by “legacy” components that bind it to restrictive methods that cannot react to sudden changes in marketplace demands.

For time-based analysis, the AI-driven environment might depend on the following: (1) continuous time analysis and (2) discrete time analysis. Each of these specific methodologies in mathematics, networks and analysis have subcomponents that become applicable to many elements in media, business/financial forecasting, signal and process management and system modeling (both complexity and accuracy).

Signal processing involves elements such as signal data visualization techniques, preprocessing and filtering techniques, plus physical-based time-domain and frequency-domain analysis (especially in real time), and the use or application of the data derived from signal processing.

Broadly defined, signal processing is a fundamental discipline in data science that deals with the extraction, analysis and manipulation of signals and time-series data. The depth of this science can get very complex and highly dependent, which is why we may see “data scientist-engineer” as a profession grow rapidly in the workplace.

Reading the Signals
In data science, a signal is defined as a gesture, action, element or sound used to convey information or instructions. From a third-person perspective, signals “transmit” information (such as instructions) by such means—i.e., by gesture, action or element/component, including audio/visual elements such as sound, light or even temperature changes in the environment, etc.

When placed into the context of signal processing, a “signal” can be any form of information that varies over time or space. Such signals may take many familiar forms, ranging from audio waveforms and temperature readings to financial market data and sensor-activity measurements. AI operations function on categorizing such signal data forms and learning the variations or changes from actual environments prompted by stimuli generated by external components including humans, the climate, physical alterations or altercations and such.

Neural Network
According to IBM, a neural network “is a machine-learning program, or model, that makes decisions in a manner like the human brain.” In our cases, these networks are specifically a computer system modeled on the human brain and nervous system, i.e., the “ideal” AI-environment. By using processes that mimic how biological neurons work together to identify phenomena, the model can weigh options and arrive at conclusions.

Conclusions are generally reached by using a series of training exercises which in turn “machine-learn” to improve their accuracy over time. As these successive training exercises are fine-tuned for accuracy (and application), they become powerful tools in data and computer science and, in turn, support artificial intelligence. The results are that tasks, such as image or speech recognition, can take seconds compared to the hours a human might require using manual identification methods.

One of the best-known examples of a neural network is Google’s PageRank (PR) search algorithm, used to rank web pages in its search-engine results tabulations. We note that PR is named after both the term web page and Google’s co-founder Larry (Leonard) Page—associated with the co-founder of Google, Sergey Brin.

ANNs and SNNs
Neural networks are sometimes cataloged as artificial neural networks (ANNs) or simulated neural networks (SNNs). There are several types or forms of neural networks, two of which are discussed here.

Artificial neural networks (ANNs) are a type of machine-learning algo­rithm that employs artificial neurons—a network of interconnected nodes (see Fig. 1 for a conceptual node diagram). These nodes then attempt to model the human brain’s neural network. Each individual node acts like its own linear regression model—composed of weights, a bias (i.e., a “threshold”) and an output. Linear regression models predict the value of a variable based on the value of another variable (see Fig. 2 for equation).

Linear regression fits a straight-line model (or surface) that is useful to “minimize discrepancies between a predicted value and an actual output value”—the approach in the training models used in artificial intelligence solutions and appears much like the slope equation from Algebra 1 (y=mx+b). For more information on the mathematics of these principles, follow up on the details of linear equations, least squares methods, and predictive coefficients.

Typically, ANNs will be used to solve complex problems—for example, facial recognition or document summary processing. Essentially, the theory behind the ANN is the teaching of computers to process data in methodologies that mimic the human brain.

Disadvantages of ANN may include: (1) they are computationally expensive and consume massive amounts of training cycles to obtain accuracy; and (2) it can be difficult for them to perfect predictions or categorize data. At this point in time, generative AI (one of the more familiar AI activities) can be considered “experimental” at best—with improvements being made as persistence in applications and libraries of training models are created.

A simulated neural network (SNN) is simply another name for an artificial neural network (ANN), considered a subset of machine learning. Summarily, ANNs are made up of connected nodes, or artificial neurons, that are loosely based on the neurons in the brain.

Discrete and Continuous
In our AI category, “continuous time analysis” refers to studying systems where changes occur smoothly over an uninterrupted time interval. Contrary to “continuous” is “discrete time analysis,” which examines systems where changes are only observed at specific, discrete points in time, essentially treating time as a series of intervals rather than a continuous flow.

The nature of the problem to be solved and the degrees (amount, type and frequency) of data being analyzed are determining factors when using either the discrete- or continuous- time analysis approaches.

Discrete time models often are the more preferred choice due to computational ease, however, continuous time models can provide a more accurate representation of certain real-world phenomena when applicable (e.g., in self-driving vehicles or other autonomous activities.)

AI Art and Approaches
AI allows leaders of organizations to make better decisions by using the built-in methodologies employed in many conditioned AI approaches to problems (such as linear regression techniques.)

Furthermore, better insights into the solution may be accomplished by uncovering patterns and relationships that others might have previously seen and thought they already understood. Fundamentally this is how AI is utilized in the generation or modification of art and images—referred to as “AI art.”

AI art is “any kind of image, text, video, audio or other kind of digital artwork produced by generative AI tools.” Such tools leverage millions of written, visual or aural content samples in reference to the prompts or known images employed when creating AI-generated art. AI art is currently integrated into many, if not all, of the major products from companies including Adobe, Microsoft, Google and more.

Those with any degree of following in media—social media and otherwise—have likely been trying to understand what this “new” technology known by the term “artificial intelligence” really means, where it was derived from, where it is headed and whether it yields another level of risk (human or otherwise).

In 1956, a small group of scientists gathered for the Dartmouth Summer Research Project on Artificial Intelligence—this is often said to be the birth of this field of research. Succeeding semiannual conferences on AI research have continued from that point forward.

Fifty years later, to celebrate its anniversary, more than 100 researchers and scholars again met at Dartmouth for AI@50, a conference formally known as the Dartmouth Artificial Intelligence Conference. This conference not only honored past and present accomplishments, but also helped seed ideas for future AI research.

Organizer John McCarthy, then a mathematics professor at the college, said the conference was “to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”

Professor of Philosophy James Moor, the director of AI@50, said the researchers who came to Hanover, New Hampshire, 50 years ago thought about ways to make machines more cognizant and wanted to lay out a framework to better understand human intelligence.

Attendees of that workshop became the initial leaders of AI research for decades. Many predicted “machines as intelligent as humans would exist within a generation.”

100 Years Hence
Even before the Dartmouth AI Research projects, as early as the early 1900s, there was media created that centered around the idea of artificial humans. Scientists of all sorts began asking, “might it be possible to create an artificial brain? [sic]” Some creators even made some versions of what we now call “robots,” the word coined in a Czech play in circa 1921.

Most early robots were relatively simple, either steam- or pneumatic-powered for the most part. Some robots could make facial expressions and even walk. In 1955, the film “Forbidden Planet” featured the robot character with AI-like capabilities and by 1963, a “real” robot (played by Ray Walston) debuted in the CBS sitcom “My Favorite Martian” and anchored the idea of a more “human-like” robot with a personality and a purpose.

Between 1950 and the mid-1950s, interest in AI really came into being. Publications by scientists would create proposals that would consider the question, “Can machines think?” The first part of this consideration should begin by defining the meaning of the two terms “machine” and “think.” We note that this consideration is today commonly referred to as machine learning (ML)—a critical part of the primary element(s) in AI.

In the early years, AI definitions could have been framed to reflect the normal use of the words “artificial” and “intelligence,” but such a seemingly narrow attitude can be dangerous—a position we are facing today with the almost inexplicable use of the AI term being applied to anything and elevated to false impressions and/or mistrust.

Computer Intelligence: Machine ‘Learning’ or ‘Thinking?’
Philosophically, the meaning of the words “machine” and “think”—when connected or used together—make it difficult to avoid the conclusion that both the “meaning” and the “answer” to the question “can machines think?” are one in the same. Definitions should not be developed just by adding another question into the answer.

Current AI solutions are often developed by repeated sampling and analyzing closely related solutions, and then making adjustments to the testing results (via algorithms) in such a way that the solution is developed by “honing in” on (i.e., sharpening) an answer that will not result in relatively ambiguous words.

In Alan Turing’s work “Computer Machinery and Intelligence” (which eventually became known as “The Turing Test,” circa the 1950s), experts could begin to use the methodologies to measure computer intelligence. Given that computers were essentially construed to be intelligent, another term was suggested (machine learning) and adopted to make the phrase a bit less ambiguous while leaving the solution set robust enough to allow improvements on the intended conclusion without leaving the actual solution “ambiguous.”

From observations like these, the term “artificial intelligence” was coined and over time came into popular use. Today, if AI’s meaning is thrust into a search engine, one might find the answer to be “a set of technologies that enable computers to perform a variety of advanced functions”—ambiguous to some degree, but still relatively understandable.

The Imitation Game
In 1950, Alan Turing explained in “Computer Machinery and Intelligence” a proposed means to test machine intelligence through an artificial game called “The Imitation Game.” The game uses three players, each isolated from one another, one of whom is an interrogator. The interrogator attempts through questioning to find out the gender (male or female) of the other two players. Interrogation is done in written (textual) form only so that voices do not give away the obvious answer. Questions continue until a conclusion is drawn about who amongst the two players is who, i.e., who is male and who is female.

This kind of testing is not unlike what happens in a modern AI environment—whereby questions are posed and answers (“data”) collected until sufficient information can point to an appropriate conclusion (aka an “answer”) to the problem. In Turing’s game, which later became “The Turing Process,” the next step changes the players somewhat by replacing one of the two “human players” (say the woman) with a machine (“computer”). The computer is then charged with simulating the substituted female human player.

Now the interrogator is challenged to find questions which can be answered honestly and accurately by both the initial (male) human or substitute (female) computer—and then from the data collected, determine which is the computer by using sufficient data and developing a proper conclusion.

IBM and Checkers
If the computer can imitate the replaced (in this case female human) elements; then ultimately by demonstration this drives the answer to the other (initial) question “Can Machines Think?”—the computer becoming the artificially intelligent “female” player.

Historically, a computer scientist named Arthur Samuel starting around 1952 developed a program to play checkers, which is the first recognized computerized application to ever “learn” the game independently.

Samuel’s programs were played by performing a “lookahead search” from each current position, essentially using the data to intelligently “predict” and test the outcome for all the remaining steps which might—or would—be played out by the pair of opposing players. Samuel first wrote the checkers-playing program for the IBM 701 and had his first “learning program” completed in 1955, which was later demonstrated on television in 1956.

Rote learning (“rote” being a memorization technique that involves repeating information until it Is remembered) and other aspects of Samuel’s work strongly suggest the essential idea of temporal-difference learning—that the value of a state should equal the value of likely following states. Samuel created this “learning by generalization” procedure which is used to modify the parameters of the value function to approach a testable and useful conclusion to the end, without ambiguities.

The entire fundamental basis for AI (i.e., “generative AI”) is built from this very simple process. Today, generative AI can be used to solve problems and can now even create new content such as images, videos, text, music and audio.

When applied to other practical purposes, it can be used to reduce costs, personalize experiences, analyze risk mitigation and apply to sustainability. However, there are drawbacks, as in any technology. Shortcomings may be the results are not accurate, may be biased (per the slant put into the testing algorithms used), may violate privacy or may infringe on copyright.

We’ve covered a bit of ground on this historic introduction and methodology summary, and in the future, we’ll dive a bit deeper into risk and management of AI—especially when it is applied widespread and uncontrolled.

TEL AVIV, Israel—Using data from the analytics tool Buzzsumo, Walkme.com has compiled a list of the most discussed technologies online in 2021, with artificial intelligence topping the list with 175,837 online articles. The closely related topic of machine learning was second, with 103,508 articles posted online between January of 2021 and November of 2021.

The buzz around AI was so high that the 11 month total amounted to about 526 articles a day. 

Technologies offering new realities also ranked high, with virtual reality holding the third spot (64,509 articles) followed by augmented reality (34,632 articles) in fourth place.

“With the pervasiveness of AI, it’s no wonder it’s one of the most discussed technologies today,” said Rafael Sweary, president and co-founder of Walkme.com in a statement. “Take another widely discussed topic, the Great Resignation, and sure enough, there’s an AI use-case tied to it. AI can help understand how humans interact with software and proactively recommend ways to improve the user experience, with actions that can be taken immediately. It’s a win for businesses, who can glean valuable data regarding technology usage and understand where the end-users are having issues. And it’s a win for the employee, who can quickly navigate the company’s tech stack, and not have to struggle with onboarding or training - especially remotely. Everything is done automatically, powered by AI and machine learning to extract data.”

The full Walkme.com top ten list of the most discussed technologies ranked by the number of online articles, is: 

1. Artificial Intelligence (AI) (175,837 stories)
2. Machine Learning (103,508)
3. Virtual Reality (VR) (64,509)
4. Augmented Reality (VR) (34,632)
5. Quantum Computing (32,548)
6. Cloud Computing (28,874)
7. Robotic Process Automation (RPA) (27,435)
8. 5G (21,928)
9. Digital Twins (9,486)
10. Edge computing (7,433)

Source: Walkme.com

More information on Walkme.com is available here.  

SANTA CLARA, Calif.—Even though revenue from global markets covers half or more of the costs for major U.S. movie and TV productions and many streaming services are rapidly launching around the world, companies continue to struggle with the difficulty of making their content appeal to the cultural diversity of global audiences. 

To help with that problem, the global entertainment technology and data company, Spherex is launching a new product called SpherexGreenlight that combines machine learning and human curation to capture societal and cultural cues from more than 120 countries around the world. 

These insights are then applied to movies and TV shows set for global distribution so that studios and networks can easily adapt their new releases to be fully compliant with local norms and reach larger audiences, the company said. 

“SpherexGreenlight was developed in response to market demand for combined cultural and regulatory insights that enable our media and entertainment clients to make data-informed decisions earlier in the production cycle.” said Spherex CEO Teresa Phillips. “Media and entertainment companies spend tens of millions of dollars annually producing new content for worldwide distribution. Greenlight assures that the content will reach the right audiences in each local market and avoid cultural missteps or censorship.”   

The SpherexGreenlight technology highlights which scenes to edit in order to comply with local regulations; helps predict how a movie or TV show will be perceived by audiences worldwide; delivers data to make informed decisions about local marketing; and helps minimizes risk of offense or censure in local markets, the company said. 

To learn more visit www.spherex.com.

https://spherex.com/

I will never forget the day I brought an Avid ProTools system into our studio and my partner remarked that “there was no way a computer could be faster than the old-fashioned razor blade edit.” There were two computers in the studio: one for accounting and the other was a crude device that controlled the capstan motor on the 24-track tape machine to synchronize it to a video machine and timecode.

This was the early 1980s; there were no computers in the OB vans and every single piece of equipment was analog. Videotape editing was machine-to-machine with an operator—the video going through a switcher and audio going through the mixing desk. Music was played off of the NAB Cart, a magnetic-tape sound-recording format. I guess you could call the first “computer” I remember in an OB van was the DigiCart instant playback system from hard drives.

After several decades of computerization and the implementation of IP throughout broadcast ecosystems, innovation has put us in a place where everything is computerized and we are already seeing the concept of computers controlling computers. Computers controlling computers is nothing new, but machine learning is, and to me this is a haunting reminder of Kubrick’s “2001: A Space Odyssey.”

SOUND AS AN INDICATOR

Artificial intelligence (AI) has been used in sports for awhile. At Wimbledon, for example, the computer listens and watches the tennis match and identifies exciting indicators by applying a variety of metrics. The metrics guide the computer in learning how to recognize significant points of interest and for what makes a good highlight or replay.

Interestingly, sound is a leading and reliable indicator. For example, pandemonium in the crowd after a long quiet pause is a good indicator of a memorable highlight moment. One of my logic metrics would also include the duration of the crowd burst as well as the amplitude, threshold, attack and sustain of the sounds during the interesting moment.

Additionally, the voice inflection of the crowd—sustained screaming as opposed to a sigh of dismay that dies out quickly is another valuable and identifiable metric. From these simple learning indicators the computer within a dozen repetitions or even 100 times will be able to accurately predict a good highlight moment.

I would argue that in 2018 we were close to something with AI. Lawo had developed a mixing system that takes data of the ball position (or any interesting follow target) and translates that into capturing the best possible sound from the optimal microphone or combination of microphones, plus determines what level to mix and blend them together. Tracking the ball is done optically and in a sport like football, the focus of the game is the ball—basically you tell the computer to follow the ball.

Undeniably World Cup 2018 was the best-sounding football event I have ever heard. Praise goes to HBS Christian Gobbel, Felix Kruckles and the Lawo team for implementing a true paradigm shift in the world of sound for broadcasting, but I think Philipp Lawo is on to something else.

THE SALSA ALGORITHM

An alternative and interesting method to advance automation is "Spatial Automated Live Sports Audio," which uses existing shotgun microphones around the pitch to detect the ball kicks. The system not only looks for overall level intensity, but also the envelope across a range of frequency bands for each sound event type that a sound mixer might want to capture. The SALSA algorithm is capable of detecting ball kicks that are virtually inaudible on the microphone feeds and is more reliable at recognizing sound events than our ears.

During live production, SALSA uses one of two approaches: It can automate a mixing console’s faders to capture each on-pitch sound event, or use the frequency/envelope information of the ball kick to trigger pre-produced samples. These sounds can be added to the on-pitch sounds or can replace the game sounds if you want it to sound like an EA Sports Game or a Saturday afternoon match on SKY. It is up to you as the sound designer and consumer.

Now, let’s take a look into another possibility of AI for sports coverage. Artificial intelligence comes into play when a computer analyzes the switching patterns of a sample of directing styles and compares the director’s commands to the position of the ball within the field of view of the broadcast cameras. The computer archives the director’s selection for future learning.

Within a short period of time, patterns will be detected, examined and programmed into event cycles to take over the direction of the cameras. A basic “follow-the-ball” pattern is learned, however it would seem possible that you can modify the production by blending and altering production styles. I once worked with a director that had a rhythm and repetition to his cutting style and literally repeated a dozen or so patterns over the course of a three-hour game.

I can clearly envision the day when bots and droid-computers capture, direct and produce live sporting events with little human intervention. Let’s follow the flow; camera robotics support systems have been around for awhile and there is no reason the cameras and audio cannot follow the electronic commands of a computer that is following play action.

Imagine this possible scenario: The computer is calculating that, after a goal kick, seven out of 10 directors would cut to a wide shot while optical position tracking is continually sending the directoid mapping data of the field of play. The “directoid” directs camera X, Y and Z to follow the ball while simultaneously directing camera A and B to track the coaches.

Additionally, cameras A and B are capturing the audio from the coaches and sending the information to the directoid, which is learning the patterns of the coaches and when to cut to the coach. The directoid has a library of possibilities for each ball position and makes comparisons.

Real-time action coverage could include speech interpretation played out from a computer that has ingested all the data and artificially created the commentary track. Speech synthesis has existed for awhile and once you have optical tracking it becomes conceivable that you can create droid commentators that interpret the play-by-play action and sound resynthesis to complete the entire experience—alternative reality.

My vision of the future paints a different picture of the science, art and practice of audio as I/we know it, but I believe my speculation could become reality.

Dennis Baxter has spent over 35 years in live broadcasting contributing to hundreds of live events including sound design for nine Olympic Games. He has earned multiple Emmy Awards and is the author of “A Practical Guide to Television Sound Engineering,” published in both English and Chinese. He is currently working on a book about immersive sound practices and production. He can be reached at dbaxter@dennisbaxtersound.com or at www.dennisbaxtersound.com.

LONDON—Media and entertainment enterprises are seeking automation to drive efficiencies with artificial intelligence and machine learning the technology keys. In a post-pandemic world where a remote and distributed work model is the new norm, AI engines could come into their own. 

“We’re at the beginning of a golden age of AI and ML,” says Hiren Hindocha, cofounder, president & CEO at Digital Nirvana, a Fremont, Calif.-based developer of media compliance technology. “The use in media is tremendous. It makes content searchable and translatable into multiple languages, allowing content to be consumed by users anywhere.” 

Others strike a note of caution. “AI/ML is working and has not led to the mass layoffs that some feared,” says Tom Sahara, former vice president of operations and technology for Turner Sports. “But nor has it reduced budgets by a huge percentage.” 

Julian Fernandez-Campon, CTO of Tedial, agrees, “AI/ML has become a ‘must-have’ feature across all technologies but we have to be quite cautious about the practical application of them. They’ve proved good results in some scenarios but in reality, broadcasters are not getting big benefits right now. Being able to test and select the proper AI/ML tool quickly and cost-effectively will definitely help adoption.” 

Roy Burns, vice president of media solutions at Integrated Media Technologies, a Los Angeles-based systems integrator, says customers are confused about what AI/ML is or what they want to do with it; “We have to explain, it’s not a magic bullet.” 

SPEECH TO TEXT

The key benefit and one ready for general operation today is captioning. Live speech-to-text quality is rapidly improving, leading to the possibility of fully automatic, low-latency creation of subtitles. Dalet reports time savings of up to 80% in delivering subtitled content for news and digital publishing. Additional ML capabilities allow systems to properly segment and layout captions to increase readability and compliance with subtitling guidelines. 

“Speech-to-text translation is probably most mature where there’s about a 90% confidence rate,” says Burns. “For some people that’s good enough but for others making an embarrassing mistake is still too risky. 

“It’s important to understand that the output of object or facial recognition tools are not human readable,” he adds. “They are designed to give a ton of metadata about the asset but to correlate it against your media you need a MAM [media asset management]. That’s what I try to explain—if you can ingest AI outputs to a MAM and correlate against a central database of record that is what is going to make it searchable.” 

Hearst Television has adapted its MAM with Prime Focus for automation of file management of commercial spots. Joe Addalia, director of technology projects for the station group, explains, “We have hundreds of commercials coming into our systems every day. If we have to mark in and out each time and do a quality check we are not being efficient. Instead, the MAM automates this, harvests necessary metadata and supplies it downstream to playout. There’s no reason for an operator to go into the item.” 

Addalia emphasizes the importance of metadata. “You can have the most glorious 4K image you want but if you can’t find it you may as well not have it. AI/ML is about being able to find what you need as you need it.” 

Hearst’s internal description for this is “enabled media:” Metadata-enriched video, audio or stills content that Addalia says will advance the possibilities for new workflow and products. 

“On top of speech-to-text, automatic machine translation is nearing maturity to enable multilingual captioning scenarios,” says Michael Elhadad, director of R&D for Dalet. “The main obstacle is that standard machine translation models are not fully ready to translate captions out-of-the-box. Automatic machine translation is trained on fluent text, and when translating each segment into individual subtitles, the text is not fluent enough; standard models fail to produce adequate text.” 

The alternative method, which consists of merging together caption segments into longer chunks, leads to another challenge: How to segment back the translated chunk into aligned and well-timed segments?

“A specialized ML method must be developed to address this challenge and produce high-quality translated captions,” says Elhadad. “This remains an open challenge for the industry and something that we’re working on at our research lab.”

CURRENT USES OF AI/ML IN MEDIA  

Additionally, object and face recognition logged as metadata can assist scripting and video editing, specifically with a rough cut being automatically created based on given metadata fields, for example, creating highlights from keywords, objects, text-on-screen. Tedial has been using AI/ML tools for some customers in the automatic logging of legacy archives, to identify celebrities and OCR using AI tools.

A big pain point is “text collusion,” when onscreen text (perhaps indicating a place or date) overlaps with caption files. Files presented with this error will immediately fail the QC of all the big streamers, but detecting issues manually in every version permutation is not cost-effective. 

OwnZones offers a deep analysis platform to scan content and compare it against other media items like time text. “The AI analysis tool can find the location of onscreen graphics using OCR [optical character recognition] and with information from time text is able to detect a collision,” explains Peter Artmount, product director for the Los Angeles-based developer of media management distribution technology. “A failure report is automatically sent back to whoever did the localization work to fix captioning before sending on to OTT services.” 

Typically, it would take an hour or two to manually QC and flag issues per hour of content; OwnZones claims its AI does it in 15 minutes. 

Another common use case is scene detection for censorship. Artmont describes a scenario in which an episodic drama containing occasional profanity requires closed captions to be blurred out for transmission during daytime hours. 

“Typically, you have to store all the versions you create, eating up storage on-prem,” he says. “In our example, you are storing 300–400 GB per show, yet the only difference between each version is to 3–4 MB of frames. We apply our AI analysis to generate a (IMF-compliant) composition playlist from the content. By storing only the differences [scenes with bad words] from the original we can trim content by 46%, making storage far more efficient.” 

CURRENT USE OF AI/ML IN SPORTS

Sports is a greenfield where the use of AI can leverage production to generate more tailored content for a specific audience. “With the reduction in live events, the ability to monetize valuable content from years of archives is key,” says Elhadad. 

There are two principal applications: Indexing (tagging, transcription, classification and object/face recognition) of vast archives; and automatic highlight creation, event-driven automatic overlays and titling. 

Aviv Arnon, cofounder of WSC Sports, an Israeli-based developer of sports media production technology, says its platform is taking “dirty” production streams of a sport into its cloud-based platform where an AI/ML system breaks the game into hundreds of individual plays. The system applies a variety of algorithms to ID each clip, make it searchable, and to clip each with optimal start and end times and then gathers the results into video packages for publishing. 

“Our ML modules understand the particular patterns for how basketball is produced including replays, camera movement, scene changes,” Arnon said. “We have all those indicators mapped to automatically produce an entire game.” 

He says sports leagues need to scale their content operations by packaging different clips to social media and other websites and that AI is the only way to do so rapidly. 

“I can’t say it’s 100% accurate but 99% is too low,” he said. “A better question might be, ‘if I had a manual editor would they have clipped it a second shorter or longer?’ It’s not about the veracity of the content. There’s no doubt it has helped speed up the process by allowing an operator to handle a lot of content in a short amount of time.” 

Elhadad explains that indexing is collected frame by frame (e.g., a frame contains the face or the jersey of a known player), but search results should be presented as clips (coherent segments where clues collected from subsequent frames are aggregated into meaningful classification). 

“While descriptive standards exist to capture the nature of such aggregation (MPEG-7), the industry has not yet produced methods to predict such aggregation in an effective manner.”

SMPTE DEVELOPING AI/ML BEST PRACTICES

Work is under way in this area. SMPTE is working with the ETC@USC’s AI & Neuroscience in Media Project to help the media community better understand the scope of the AI technology. 

“AI is promising but it’s an amorphous set of technologies,” says project director Yves Bergquist, noting that a quarter of organizations report over a 50% failure rate in their AI initiatives. Consultancy Deloitte also found that half of media organizations report major shortages of AI talent. 

“There are a lot of challenges around data quality, formats, privacy, ontologies and how to deploy AI/ML models in enterprise,” Bergquist says. “AI/ML is experimental and expensive and there are duplications across the industry. 

“We think there are strong opportunities for interoperability throughout the media industry,” he adds. “Not everything has to be in the form of standards. We also want to share best practices.” 

The task force is made up of about 40 members including Sony Pictures, WarnerMedia and Adobe. Research is focused on data and ontologies, AI ethics, platform performance and interoperability, and organizational and cultural integration. “That last topic is the most important and least understood challenge,” Addalia says. “It speaks to the ability of an organization to create an awareness and understanding of what AI is and is not, how to interpret its output, how to understand the data.” 

The task force is “expecting to make a substantial contribution in the area of suggesting best practices in terms of creating a culture of analytics,” he said. 

Addalia agrees that no single tech provider can do it all. “TV is a cottage industry,” he said. “We have to use our collective resources properly so we can leapfrog into next-gen where we have automated workflows. This requires cross-industry collaboration. Vendors must work together.” 

There’s also an onus on end users to provide feedback, Addalia added. “They need to define the use case and the desired results,” he said. 

Muralidhar Sridhar, vice president of AI/ ML products at Prime Focus says M&E needs specific treatment and that ready-made AI engines have not transformed the industry. 

Human-like comparison of video masters, transcription for subtitling and captions, AI-based QC, reconformance of a source from a master, automatic retiming of pre- and post-edit masters are all important use cases, he says, provided the AI can be made to understand the nuances. 

“The problem is that while most AI engines try to augment content analysis, the ability to accurately address nuances is not easy to match,” he said. “Marking segments like cards, slates and montages takes time and cost. It needs 100% accuracy and 100% frame accuracy." 

Prime Focus’ prescription for clients like Star India, Hotstar and Hearst Television is to combine computer vision techniques with neural networks where necessary then customize the solution for the customer. 

DEVELOPMENTS AHEAD

Higher accuracy and speed across all AI/ ML capabilities continues to advance. Fernandez-Campon advocates “Intelligent BPM,” where AI tools improve workflows by taking automatic decisions and focusing the work of operators on creative tasks. “That’s why it’s key to offer flexible and cost-effective AI/ML tools that can easily integrate into workflows and be swapped one for another,” he said. 

Dalet is working on a new approach to indexing faces and creating “private knowledge graphs” it thinks will speed up the process of cataloging large content archives and building libraries of local personalities for regional needs. Current technologies are not able to easily recognize faces throughout a historical archive, the company says. 

“Better context-aware speech transcription, more accurate tagging and smoother automatic editing will further reduce the need for manual, repetitive tasks,” says Elhadad. “AI/ML empower content producers and distributors to focus on higher value creative work. AI/ML is becoming pervasive: another tool in the box that users will find increasingly beneficial without the doom-laden ‘man vs. machine’ context.” 

SAN FRANCISCO—Zorroa Corp. has launched Boon AI, its machine learning (ML) software-as-a-service (SaaS) platform, which cuts the cost of ML adoption and makes machine learning more accessible for media organizations.

Available in the Google Cloud Marketplace, Boon AI enables media technologists to integrate ML capabilities into their digital media supply chain via APIs in days or even hours. With a single point-and-click visual interface, Boon AI drives workflow automation and opens new revenue streams not feasible without AI and machine learning, it said.

“We’re excited to partner with Zorroa and to help deliver its machine learning integration platform on Google Cloud,” said Kip Schauer, global head of Media & Entertainment Partnerships at Google. “With just a few clicks, customers can deploy Boon AI on the Google Cloud Marketplace to help break down the machine learning adoption barrier and enable workflow efficiencies and new revenue streams.”  

Boon AI modernizes media supply chain workflows by making it possible to kick off ML projects without code in under an hour, the company said.

It supports multi-vendor interoperability with direct access to the ML ecosystem, which includes Google Cloud, AWS and Azure ML APIs, and can seamlessly integrate ML-generated metadata into media management and production applications without breaking existing workflows, Zorroa said.

Boon AI scales rapid-cycle innovation by giving media technologists the tools needed to build ML-powered applications with the same agility of traditional software development, it said.

Just as no-code software allows for app development without programming knowledge, Boon AI allows for ML API integrations without dedicated development or data science teams, the company said.

Zorroa is a provider of accessible machine learning integration solutions. It is backed by Gradient Ventures, Google’s Ai-focused venture fund.

More information is available online.  

For many different reasons, 2020 has been a year we will never forget. Global lockdown orders have elevated viewing figures for all media to unprecedented levels, with subscription to and streaming of OTT services a particular beneficiary. More content has been watched by more people than ever before, but what are some of the trends behind the headline figures, and what can they tell us about the way that the industry has rapidly evolved during the pandemic?

From our conversations with our customers and partners, and being involved in industry research projects around the world, we have identified the six following trends that will have a key impact in 2021 and beyond.

THE IMPORTANCE OF CONTENT

Increasingly we are seeing that companies both create and distribute content, with very few either sticking to solely creation or distribution, but rather taking total ownership of the entire vertical.

We have seen an increasing move towards live content as the year has unfolded. Despite 2020 being a dreadful year for live events—with sports particularly affected as seasons paused and then truncated—live entertainment has fared little better and we are starting to see live content featured more heavily in the plans of streaming services, and even overtaking on-demand in some cases. 

This is almost a complete reversal from the start of the pandemic, and live linear is a very large part of this movement, as witnessed by the launch of services such as NBC Universal’s Peacock, Fubo’s expansion and successful IPO, as well as the growing interest from OTT providers in offering local feeds to their subscribers.

AN INCREASE IN VIRTUALIZATION

We are seeing a big growth in software-defined streaming infrastructure versus hardware-defined infrastructure.

This trend goes back over a decade but seems to be accelerating in the pandemic as people move urgently toward remote production workflows and are therefore having to implement plans that they have already had on the drawing board for some time. Currently, the reliance on software-defined vs. hardware-defined infrastructure is hovering around a 65:35 split, but we expect this to shift further to a 90:10 ratio over the next few years.

DELIVERY OVER IP

We are a long way from the time when broadcast video was delivered over finite types of connectivity. Fifty-six percent of the respondents in a recent survey on the state of the streaming industry are now saying they will be using standard internet connections over the next two years, with the rest comprised of 25% CDN, 12% fiber, 4% cellular and 2% satellite. 

As a caveat, the survey did not allow for multiple answers, but we were surprised to see that the internet was that high and conversely cellular was so low.

THE GROWTH OF 5G

The 5G introduction to the consumer market has been complex, but in terms of B2B usage it is already starting to make waves, and as a result we’re conducting active production pilots with major broadcasters in partnership with Verizon and AWS Wavelength Zones. The first workflow that we are putting into production is distribution, effectively a satellite-replacement distribution workflow over 5G.

Distribution is exciting work and seen across the industry as a key use case for the future of 5G, followed closely by direct-to-consumer applications. Contribution, B2B uses and in-venue delivery are all gaining interest but not quite yet building up significant momentum. We suspect that contribution specifically will end up becoming significant in the coming years.

What level of impact will 5G have on video streaming strategies? Over the next two years opinion is mixed, with the industry largely spread out on a scale between a total game-changer or having no impact at all. Though if anything, opinion is slightly weighted to the latter end of that scale.

THE NEED FOR INTEROPERABILITY

Unsurprisingly given the industry’s increasing path to complexity, the majority of companies are stating that interoperability is of the highest importance to their streaming infrastructures.

This is partly a result of people doubling down on their existing infrastructures and not wishing to throw everything out with the installation of a new system and makes perfect sense given the level of mergers that have dominated the media industry, as well as modern production workflows that touch far more than the single organization. No one wants to find themselves trapped in a silo.

THE IMPACT OF MACHINE LEARNING

The streaming industry sees quality of service as being the primary use for machine learning (ML) deployments, with predictive maintenance and transport optimization tied for second place. Anomaly detection and security are distant considerations, though arguably all of these are related to QoS in one way or another. 

As shifting workflows within and between media companies increase in complexity, the data and analytics gathered from across that supply chain can be mined and interpreted using AI and ML tools to ensure optimal Quality of Transport, Quality of Service, Quality of Content and therefore create optimal Quality of Experience for end users.  

We see potential in using AI and ML to streamline broadcast workflows even further, including better identifying root cause analysis, optimizing video transport across different IP networks and much more. We’ve also used ML to work out that there are situations where we shouldn’t be using ML, and that some things are much better being hard-coded in. In other words, there are valuable things you can learn from ML even if you don’t put it into production.

One thing ML can’t do is predict exactly what trends we’ll be talking about as 2021 unwinds. At this time a year ago few among us could have foreseen the impact of COVID-19 on the industry and the rest of the world. But these six trends represent issues are the starting point, and it will be fascinating to monitor how they develop over the next 12 months.

John Wastcoat is senior vice president of strategic alliances and marketing at Zixi.

Over the years, the complexity of video preparation and delivery has increased dramatically. First, the industry witnessed the move from tape to file-based workflows, followed by the transition from analog to digital. New formats and standards have also emerged, adding to the complexity of video delivery. 

Aside from these technology transformations, consumer viewing habits are shifting. Today’s viewers prefer OTT media services, with 76% of U.S. households subscribing to OTT services compared with 62% for traditional pay-TV, according to the latest research from Parks Associates. As broadcasters deliver a higher volume of content to a wider range of screens and global audiences, additional errors are being introduced into the workflow, potentially affecting video and audio quality.

Recent advancements in automated media quality control and monitoring systems are helping broadcasters deliver error-free video and audio on every screen. In particular, innovations in machine learning and artificial intelligence are pushing media QC and monitoring to the next level, increasing the accuracy and consistency of certain media tasks, including content classification, content categorization, lip sync checks and more.

MEDIA QC AND MONITORING IS EVOLVING

In the early stages of media QC and monitoring, automated systems were limited to simple tasks, such as checking the correctness of audio/video technical parameters, including resolution, frame rate, bitrate, content structure and container parameters.

Since then, media QC and monitoring has evolved. Today, broadcasters can check for perceptual errors using computer vision and standard audio processing techniques. These checks include interlace artifacts, defective pixels, dropouts, visual text recognition, compression and ghosting artifacts, loudness and language detection. 

With the rise of ML and its success in completing tasks such as content classification and object detection, the scope of media QC and monitoring has expanded. Now broadcasters are using advanced ML techniques capable of semantically understanding content for the purpose of content moderation, content classification, indexing and description generation. Let’s look at a few of the specific media applications that can be optimized with ML and AI technologies.

SPEEDING UP CONTENT COMPLIANCE WITH ML 

Monitoring and altering content in order to conform to different rules and regulations is one application that can greatly benefit from ML. Broadcasters must comply with a wide range of rules and regulations, which can vary from one region to another. 

Traditionally, broadcasters have maintained a pool of human moderators to manually filter content for regulatory compliance. Under a typical manual workflow, content is passed through multiple stages of review. If a review fails at any stage, the content goes back for editing. Manual content QC and monitoring is expensive, time-consuming and inaccurate. With so many global and regional aspects of content moderation, it is almost impossible for humans to carry out the job with 100% accuracy.

By automating this process, broadcasters can eliminate the limitations of manual content moderation, including the inability for people to memorize a significant number of visual symbols and the possibility for human error. With an automated QC and monitoring workflow, broadcasters can more rapidly and accurately check content for the presence of brand names, hate symbols, alcohol, violence, celebrity faces, vulgar speech captions and religious symbols. 

When using an automated system powered by ML, computer vision techniques and computer algorithms, the benefits are even greater. ML-based systems can handle huge and multiple content classification check lists without any major performance limitations, driving broadcast workflow efficiencies. 

However, it’s important to note that while current ML solutions are sophisticated and may be combined to create broader applications, they lack the real-world knowledge and human experience needed to create valid and acceptable outcomes on their own. Human input is still required to confirm the validity of patterns and help machines refine the result. Such human interactions are likely to define ML uses in the media industry for the foreseeable future.

ENSURING SUPERIOR QUALITY CAPTIONS WITH ML

Checking for the presence and accuracy of captions is another application area where ML has proven to be very effective. ML can be used to automatically generate captions where they are not present in the content, check the alignment between captions and audio, and check the correctness of the captions against the spoken audio. In addition, ML simplifies the identification of speakers in audio, ensuring that the correct punctuations are placed in captions. 

Ultimately, with ML, broadcasters can expedite the caption creation and verification processes for both live and VOD content, while ensuring that when content is delivered in multiple video quality levels within OTT video streams, the captions maintain a high quality.

Over the last decade, automatic speech recognition engines have achieved extremely high accuracy, up to 85%, via ML. Still, automatic speech engines face several challenges, such as robustness issues in noisy environments, the ability to handle variable accents, problems when multiple speakers are talking at the same time, and difficulty with kids’ voices (due to a lack of data to train ML models).

Keeping humans in the loop is imperative to resolve these challenges. By combining cutting-edge ML and automatic speech recognition technology with a manual review process, broadcasters can bring increased simplicity and cost savings to the creation, management and delivery of captions for traditional TV and video streaming.

ELIMINATING AV LIP SYNC ISSUES WITH ML

Synchronization between audio and video is a common issue today. Leveraging image processing and ML technology and deep neural networks, broadcasters can automatically detect audio and video sync errors. ML offers a faster and more precise approach to detecting audio lead and lag issues in media content, compared with the traditional approach of manually checking for lip sync errors. This allows broadcasters to provide a high quality of experience to viewers (QoE).

Through the power of ML, broadcasters can perform facial detection, facial tracking, lip detection, lip activity detection and speech identification. With an ML-based lip sync solution, typically one module uses video to extract faces and track lip movement. A second module  uses audio to extract audio features and a third ML module matches the movements with the audio features. Using this technique, it is possible to detect even one frame of synchronization issues.

CONCLUSION

The amount of content that broadcasters are delivering across the globe is massive. Ensuring a high-quality video experience on every screen is essential if broadcasters want to keep viewers satisfied. With automated QC and monitoring solutions featuring ML and AI technology, broadcasters are better placed to quickly and more accurately comply with industry and government regulations, deliver high-quality captions, classify and categorize content and eliminate lip sync issues. 

Anupama Anantharaman is vice president, Product Management, at Interra Systems.

MILAN, Italy—As the world of sports broadcasting is evolving, international system integration consultancy Fincons Group has released a new whitepaper that details the new technology that is essential to keep up and how exactly it can be properly utilized.

“Next Gen Technologies Now at Bat: Machine Learning, AI and Next Gen TV Drive the Sports Entertainment Experience Revolution” provides an insight into how technology like machine learning and next-generation TV standards like ATSC 3.0 and HbbTV will usher in a new era of personalization and interactivity—put on the fast track due to the coronavirus pandemic that is leaving sports broadcasts without one of their staples, live fans.

These new technologies will enable sports content owners to leverage data in new ways, including for targeted, personalized advertising and monetization with access to near real-time data analysis and prediction capabilities that can be applied to both the sporting action and audience preference and behavior.

The white paper focuses on four key areas: data analytics, gamification, smart watching and targeted advertising and ecommerce.

“It is critical for this forward thinking industry to remain constantly poised toward the future by harnessing these new technologies and introducing an expanded Sport Supply Chain with production-ready software frameworks to accelerate transformation ahead of competitors,” Fincons said.

The “Next Gen Technologies Now at Bat: Machine Learning, AI and Next Gen TV Drive the Sports Entertainment Experience Revolution” is available for free download here. 

SUNNYVALE, Calif.—Fox Sports is working with Google Cloud on a new system that will help automate the logging, discovering and storing of the network’s video assets by utilizing Google Cloud’s video search and advanced machine learning capabilities.

With Google Cloud’s open architecture and advanced technologies, Fox Sports anticipates to be able to organize, discover and build on petabytes of data and millions of videos it has in its collection, as well as the new content that is being produced daily.

Google’s machine-learning models are being used by Fox Sports to automatically identify and label clips across different categories. The technology also incorporates a user-friendly interface for faster discovery of clips.

“We’re excited to team with Google on this project that will allow Fox Sports to take the next step toward the most state-of-the-art workflow for accessing all of our media,” said Brad Zager, executive producer, executive vice president/head of production & operations at Fox Sports. “It’s been amazing to see all the different tools that Google Cloud will provide our network to transform the way we work, while also helping to make the way we create content so much easier for our production teams.”

OTTAWA—Trend alert: Artificial intelligence/machine learning (AI/ML) is becoming an integral part of the total TV production/playout process.

“AI/ML is shifting to provide tremendous value to broadcasters and content producers,” said Amro Shihadah, IdenTV’s Co-Founder & Chief Operating Officer for IdenTV, a McLean, Va.-based real time video analysis market researcher. “AI/ML is achieving this by transforming big data from a cost center and opaque set of structured/unstructured datasets into real-time actionable analytics and tools for big data search and recall, creating a better user experience, and generating revenue from new content distribution channels.”

Broadcast consultant Gary Olson, who has just released the second version of his book, “Planning and Designing the IP Broadcast Facility—A New Puzzle To Solve,’’ says the technology is already showing up in elements of the production chain and is expected to expand its footprint.

“I see AI/ML appearing in editing, graphics and media management products in 2020,” Olson said. As the year progresses, “it will be interesting to see which vendors will claim their products have AI or ML.”

CONTENT DISCOVERY

Many major broadcasters and TV studios have vast libraries ripe for direct-to-consumer online sales. The challenge lies in determining which of these programs will appeal to modern consumers and for what reasons, without using employees to watch all of them in real-time.

Prime Focus Technologies’ CLEAR Vision Cloud has a cloud-based AI engine that can do this work across a number of search variables, and in “record time,” according to the company.

“There could be one AI engine that looks at identifying faces in the video,” said Muralidhar Sridhar, vice president of AI and Machine Learning for PFT. “Another one may look at signature sounds of, ‘let’s say, a person splashing through water,’ while a third searches for distinct objects. Best yet, what would take humans hours to achieve looking at a piece of content can be done by our AI in real time.”

Primestream’s Xchange platform uses AI/ML to power its content discovery tools, providing a wide range of search options in the process, according to Alan Dabul, director of product development for Primestream.

“You can narrow the search down not just to President Trump, but to those specific clips where he is talking about taxes,” he said. “You can then narrow the search further to those times when he is speaking about taxes in an office setting, and then see who is with the president in the shot at that time.”

SPORTS AND LIVE EVENTS

Sports and other live events are among the most labor-intensive productions for broadcasters, given how much content has to be created on the fly. Tedial’s SMARTLIVE metadata engine uses AI/ML to automate media management tasks associated with these productions; including metadata tagging, automatic clip creation and distribution during live events to digital platforms and social media. SMARTLIVE can also manage multivenue feeds and support multiple, instantaneous content searches to integrate archival footage into live broadcasts.

“SMARTLIVE allows the production team to create more content leading to increased fan  engagement and additional revenue, using the same budget and with the same team,” said Jerome Wauthoz, vice president of products for Tedial. “SMARTLIVE also connects directly to existing production environments so our customers can use their current infrastructure to ingest, edit and deliver content; no additional investment is necessary.”

CAPTIONING AND TRANSLATIONS

Another labor-intensive area where AI/ML is gaining traction is multilingual captioning. Using speech-to-test AI systems, vendors can automatically generate text captions from the content’s audio, and provision them in a range of languages within the same data stream.

“The algorithms are trained to learn from data in real-time, absorbing local terms and dialects for the optimal captioning experience,” said Brandon Sullivan, senior offering manager for IBM Watson Media. “As AI and machine learning training capabilities improve, local dialects, places and specific names, as well as the voices of individual speakers, will all be accurately captured. Down the road, this will not only transform closed captioning but also automated translation, video indexing, and more.”

Captioning and lip sync are two of the AI/ML technologies featured as part of Interra Systems’ BATON, a video QC platform. “With AI/ML, you can improve the accuracy and speed of captioning, which is a resource-intensive, time-consuming process,” said Anupama Anantharaman, vice president of product management for the Silicon Valley-based provider of video QC and monitoring technology. “It is also particularly effective at detecting ‘lip sync’; the alignment between the movement of lips onscreen and what is being said.”

Telestream’s Telestream Cloud includes captioning as its many cloud-based AI/ML-enabled offerings; the others being video transcoding for multiple delivery platforms and quality/compliance checks, according to Remi Fourreau, cloud product manager for the company.

“We use the speech-to-text capabilities of many cloud-based providers to generate accurate captions and subtitles in many languages,” Fourreau said. “This is an area where AI/ML really shines in doing the task accurately and efficiently.”

ENCO’s enCaption4 platform provides automated closed captioning for live and pre-recorded TV content in real-time, and combines AI-driven machine learning with a neural-network speech-to-text engine. In addition to newsroom rundown imports that teach unique words via AI, enCaption4 can be taught special words such as host and cast names, and local and regional terms. Other AI-driven enhancements improve the captioning of punctuation and capitalization.

“enCaption can accurately spell unusual words learned from ingested lists and scripts, and without creating speech pattern profiles for every speaker, said Ken Frommert, president of ENCO. “This is an important benefit for news operations automating and captioning speech from various anchors, reporters, meteorologists, and studio guests.”

COMPRESSION

Video compression has always been a balance between data rate reduction and video quality. Through AI- and ML-based cloud solutions such as its VOS360 Live Streaming Platform, Harmonic aims to strike this balance more effectively. 

“Our PURE Compression Engine uses AI/ML to improve the algorithms that manage video compression,” said Jean-Louis Diascorn, senior product marketing manager, who leads Harmonic’s AI/ML for video compression advances. “These improvements are achieved far quicker using AI/ML compared with using human engineers. We continue to make progress on the work that we presented at last year’s NAB BEITC and are now aiming to address the density aspect.” 

RECOMMENDATION ENGINES

Streaming services such as Amazon, Netflix and YouTube use AI/ML-enabled recommendation engines to mine their viewers’ current content choices, and use what they find to recommend similar programs that might be of interest. Vionlabs’ AI/ML-enabled Content Discovery Platform is designed to help broadcasters assess their own content libraries, to focus and enhance their Direct-to-Consumer sales online.

“High-quality data can help broadcasters understand so much more about their content and make better informed decisions throughout the content cycle,” said Marcus Bergström, CEO of the Swedish-based provider of video discovery technology. “One example of this is in content recommendations and providing broadcasters with a deeper understanding of how successful shows appeal to viewers. It could also help them automatically comply with regulations for post-watershed content.”

Last month, the company launched “Emotional Fingerprint API” to help media companies make better decisions based on AI-generated video data and insights. Emotional Fingerprint API uses computer vision and machine learning to generate sentiment-data, creating a unique personal viewer experience based on Vionlabs’ recommendation, according to the company.

Emotional Fingerprint API has been developed to measure thousands of factors during the screening of a video, including colors, pace, audio and object recognition, in order to produce an AI-derived fingerprint, frame by frame, that represents the emotional structure of content.

THERE ARE LIMITS

AI/ML-enabled systems are now fulfilling many roles in the TV production/playout stream. But they can’t do everything; at least not yet.

“For machine learning tools to work effectively, you need to continuously fine tune models and need large amounts of well-prepared data,” said Anantharaman. “There will be challenging situations where human intervention will be needed. Yet, for the majority of content, AI/ML can provide an extremely high level of accuracy.”

WASHINGTON—As we begin a new decade, the media archive landscape is going through significant changes. While the traditional requirement to preserve content has always existed, new ways to deliver content offer greater potential for monetization now than at any time in the past. New technologies and vendors have also arrived, promising new capabilities and lower costs. And of course, file sizes and shooting ratios continue to increase.

It is clear that the cloud will play a major role in the future of media archiving. Private cloud solutions, where the organization’s infrastructure is shared among multiple groups, departments, stations or regions, are used heavily today. Public cloud solutions, from providers such as Amazon, Microsoft, Google, and others, have compelling services that offer low up-front costs but a continuous operational expense that usually increases over time. Hybrid solutions that incorporate both public and private cloud, may enable customers to utilize the strengths of both private and public clouds.

It is important to note that there are a wide variety of companies offering cloud-based infrastructure, with each one providing unique products and services. Some may offer cloud-based processing, cloud storage, content processing, and machine learning services, while others may focus on fewer services. In addition, their storage tiers and pricing structures may vary widely. Differences in storage and egress costs, retrieval responsiveness, as well as the number and locations of their data centers are all factors that may be considered as part of the decision to select one vendor over another.

THE (GRADUAL) RISE OF CLOUD STORAGE

Public cloud technology has now reached the point where almost every media organization is using it in some form as part of their daily operations. While the majority of media archives are kept on-premise today, some organizations have begun to rely on public cloud storage as part of their archive solution. This can take several forms, such as:

Primary archive storage: Archived media files are stored on public cloud storage. When content is required by the organization’s users or systems, it is restored from the public cloud.

Business continuity: Similar to primary archive storage, archived media files are stored on public cloud storage. However, in this case copies of the media files are also stored on-premise and the public cloud copies are retained as a backup in case the primary storage or location is not accessible or usable.

Collaboration: Selected archived media files are stored on public cloud storage, so that they can be accessed by remote users, external contributors, or business systems.

In addition to cloud storage, some organizations are using the public cloud to host the asset or archive management application, reducing or eliminating the need to deploy servers in their data center(s). In addition to the elimination of CAPEX (capital expenditures) costs, this can also give them the ability to scale their computing resources up and down as needed in a way that would be cost-prohibitive on premise.

THE ROLE OF AI AND MACHINE LEARNING

Almost every organization has a significant amount of content that has little or no metadata. Artificial intelligence and machine learning systems offer an automated way to extract metadata from files, through services such as speech-to-text, facial recognition, object recognition, and sentiment analysis.

The accuracy of these services varies, but with companies like Amazon, Apple, Facebook, Google and Microsoft investing heavily in these technologies, it is likely that we will see major strides in their effectiveness in this decade. Consumer adoption of products using speech recognition and visual recognition is a clear indicator of the appetite for these technologies. In early 2019, Amazon informally announced that 100 million Alexa-powered devices had been sold globally, and NPR/Edison estimates that there are almost 120 million smart speakers in the United States alone. The use of devices like these, along with visual recognition tools used by OTT and social media applications, will continue to expand the size of the “data lakes” that help to refine these technologies.

Automated metadata extraction will enhance the value of media archives. By enriching archived assets with metadata, it is now possible to begin to understand what value they may have. The assets can now be exposed to both internal and external users who may be interested in using them. This will lead to the development of new tools and channels to enable consumers, companies and other organizations to search, browse, and purchase content directly from the owner/creator.

MANAGING ASSETS, NOT JUST ARCHIVING FILES

To maximize the potential of the modern media archive, asset and archive management systems need to do more than just control storage systems. They need to understand file formats, metadata and workflows, so that they can get content to the right place, at the right time, in the right format. Business rules can be used to determine the right format, wrapper and bitrate to be used for the archived asset and for any copies that may be created as it is moved through the facility. By storing metadata and the asset together in formats that can be read by other vendors, these systems can also make sure that the content is portable and future-proofed.

Asset and archive management systems are critical to the integration of public cloud storage, not only because they provide the interfaces between that storage and the customer’s production and/ or distribution systems, but also because they can manage hybrid storage environments. These environments may have a mix of on premise and cloud storage, or even multiple cloud services, and moving or migrating content between the different systems must be automated and seamless. The management systems should also provide tools that predict future costs as the cloud storage usage increases.

Solutions such as Masstech’s Kumulate can provide the tools to easily manage these processes and to ensure that your organization is ready for the next decade and beyond.

Savva Mueller is vice president, product management and marketing for Masstech.

A new poll by TVBEurope has found that people working in media technology think AI and machine learning will have the biggest impact on the industry.

As part of its State of the Market webinar, TVBE asked the audience to vote for which technology they thought would have the most impact on the media technology industry over the next few years. The options were 5G, Internet of Things, AI and machine learning, and UHD/HDR.

AI and machine learning received 52.4% of the votes cast, followed by 5G with 23.8%, IoT had 14.3% of the vote, while UHD/HDR finished with 9.5%.

The webinar also asked the audience to choose what they think will have the biggest impact on the market in the next five years. In this poll, consumer behavior was the overwhelming choice with 76.5% of the audience opting for it. Technology and mergers and acquisitions activity were tied for second place with 11.8% of the votes cast each.

The webinar featured a wide-ranging discussion about the state of the media technology market and its outlook for 2020 and beyond with Martyn Whistler, lead analyst, media and entertainment, EY; Sinead Greenaway, chief technology and operations officer, UKTV; and Google’s Head of Broadcast and Entertainment for the U.K. and Ireland, Justin Gupta.

The State of the Market webinar is available to watch on-demand.

NEW YORK—The NAB Show New York will serve as the background for Enco to showcase its latest advances with machine learning for the enCaption4 automated live captioning system and enTranslate automated live translation systems designed to help hard-of-hearing and non-native-speaking audiences.

The enCaption4 is used to automate and augment live or pre-recorded content with closed or open captions in near real time. New features on display include advanced punctuation capabilities that enhance the accuracy and readability of transcriptions for full stops, commas, exclamation marks and question marks in English, German, Spanish and French. Enco describes the system as being able to detect the context of sentences to insert the appropriate punctuation and capitalization. Other new features offered are an enhanced scheduling interface, a web API for third-party integration and additional boosts to accuracy and speed.

The same speech-to-text engine in the enCaption4 is used in the enTranslate system, featuring advanced translation technology powered by Veritone. The enTranslate platform provides subtitles and secondary or tertiary closed captions in alternative languages, while users can also embed translated captions in VOD content or show live. It features a Neural Machine Translation methodology that blends AI and sophisticated linguistics modelling to provide translations based on context of current words and phrases. There are 46 languages supported by enTranslate.

Enco will display these two platforms at its booth, N355, during NAB Show New York, Oct. 16-17 at the Javits Convention Center.

JOHANNESBURG, South Africa—Live sports remains one of the most watched elements of live television, but that doesn’t mean that the experience still can’t be improved upon. According to a new report from research and development company NTT around 54% of viewers are not fully satisfied with sports viewing experiences, but they also say that a solution could be found in the form of artificial intelligence and machine learning.

One area of interest for the use of AI in sporting events is for predicting outcomes. More than half of survey respondents age 18-34 (54%) think AI is capable of successfully predicting the results of a sporting event, with nearly the same amount (52%) suggesting that accurate predictions would make a sporting event more engaging. However, across all ages, only 26% of respondents were aware of AI or machine learning technology being used for sporting events.

Additional findings from the NTT report reveal that 56% of millennials (18-34) said they expect to increase their use of “second screens” during live sporting events over the next three years. Across all age groups, 51% of respondents are tracking live updates of sporting events through a second screen at least once a week. A key motivator for including a second screen experience, according to 34% of participants, is for greater access to data and stats.

“It’s clear ICT infrastructure, the cloud and mobile services have a critical role to play as the sport industry evolves to meet the growing demands of digitally savy supporters,” said NTT in its announcement.

LIEGE, Belgium—EVS will launch a new automated multi-camera positioning and framing system at IBC 2019, Sept. 13-17, at the RAI in Amsterdam.

The new autonomous camera system analyzes the images of robotic camera feeds in real time and guides cameras toward live action with the appropriate zoom. The product, which integrates with the company’s X-One unified production system, relies on machine learning enabled by EVS’s AI-driven VIA Mind to mimic the moves of human camera operators, the company said.

The company will also highlight new storytelling functionality for its X-One unified production system that allow a single operator to produce highlights and dynamic live productions.

EVS will feature its new centralized ingest solution that relies on its XS-NEO software-defined server and IPD-VIA ingest app. The solution offers fast, simple ingest with dynamic, concurrent support for multiple formats, codecs and frame rates. Hosted on the EVS PMR processing module and leveraging the benefits of IP connectivity, the ingest solution supports SDI-based productions and uses the company’s loop recording technology. The ingest solution uses an HTML 5 web interface to control and schedule live feeds ingested into the XS-NEO server.

The company will also introduce the latest version of its Dyvi software-defined switcher and unveil the latest XT-VIA and XS-VIA servers.

See EVS at IBC 2019 Stand 8.B90.

To learn more visit the EVS website. 

Today, content creators are increasingly looking to expand across geographies, but one of the biggest challenges they face is overcoming language barriers. Delivering content to non-native speakers is possible through captions and subtitles, or audio dubbing. Yet, quality control (QC) checks are imperative to ensure that everything flows smoothly. This article will examine why audio dubbing is better than providing subtitles, the key capabilities to look for in a QC system and how machine learning (ML) and artificial intelligence (AI) technologies are simplifying QC workflows for audio dubbing.

WHY AUDIO DUBBING MAKES SENSE VS. SUBTITLES

Subtitles are an easy and affordable way to deliver content to non-native speaking viewers, but they have limitations. For example, using subtitles, it can be difficult to deliver long dialogue scenes within limited screen times. Furthermore, subtitles can be distracting to viewers, since it makes them focus on the text vs. important details in the scene. Another reason why subtitles aren’t ideal is because they don’t express the emotions being delivered in the dialogue.

Audio dubbing to regional languages is an alternative option that translates a foreign language program into the audience's native language. Dubbed tracks are created by adding language-specific content to the original audio and have become a cost-effective solution for content creators to reach audiences in different areas of the world. When creating dubbed audio tracks, content creators must ensure high quality and synchronization with the video or original audio track.

HOW TO SIMPLIFY AUDIO DUBBING QC WORKFLOWS

Recent advancements in ML and AI technologies have simplified the way that content creators perform quality control on audio dubbed tracks, automating the process of verifying languages and assuring synchronization.

There are several key capabilities content creators should look for in an ML/AI-powered QC solution for dubbing workflows in order to ensure the utmost efficiency, accuracy and quality of experience for viewers.

One important feature is the ability to verify dubbing packages with complex structure. The QC solution should be able to verify complex dubbing packages. Oftentimes, a dubbing package consists of multiple MXF and .wav files as opposed to a single file. The MXF file contains video tracks along with original audio tracks and .wav files corresponding to multiple dubbed audio tracks. Sometimes .wav files represent individual channels of a 5.1 audio track, or there are multiple audio tracks or channels encapsulated in container formats instead of .wav files. The QC solution should be able to handle package variations and verify the multiple dubbed audio tracks properly.

For audio dubbing, metadata verification is a must-have capability in a media QC solution. The system needs to verify the number of audio tracks or the channel configuration of multiple dubbed tracks, along with the duration of the original audio track compared with the dubbed audio tracks.

Language identification is an integral component of a QC system to ensure that each track has the correct intended language. Over the years, machines have become more intelligent and computing power more affordable, making accurate, automated language detection a reality. If the content creator has access to a few hours of audio content in the target languages, they can be used to train the ML models for language prediction purposes. Content creators can then verify that the language is correct by checking it against metadata.

During the audio dubbing process, content creators must make sure synchronization between video and dubbed tracks exists. This can be challenging, as there is no way to map the correct audio sequence for any video screen with dialogue. However, the majority of video content contains black frames and color bars that are designed to help in meeting requirements like synchronization. The corresponding audio sequence for black frames and color bars is silence and test tone, respectively. Choosing a QC solution that can verify the presence of black frames in video tracks as well as silence while color bars appear with test tone in dubbed audio tracks is critical.

Synchronization must also be present between original and dubbed tracks. This is no easy feat since audio data in master and dubbed tracks are completely different. It’s likely that common background music or effects were not used for the master track and dubbed track. These can be separated from the audio track using mechanisms like band pass filter.

Checking for loss of synchronization between background beds of dubbed audio tracks and the original audio track can easily be performed by a QC system with ML and AI technology. The challenge is ensuring a proper separation of background bed from audio tracks. One way this can be achieved is by comparing loudness curves. The technology will compare loudness curves, checking for mismatch between loudness values of original and dubbed tracks.

CONCLUSION

Through audio dubbing, content creators can reach audiences all around the world, providing content in their local language without the limitations of subtitles. Automating QC dubbing workflows speeds up this process immensely, while also introducing increased accuracy. When ML and AI technologies are added to the workflow, dubbed packages can be created even more rapidly, with reduced manual intervention. It’s a win-win situation for content creators and viewers. Content creators can expand their brand into new regions of the world, and viewers are assured a high-quality television experience in their native language.

Niraj Sinha is the principal engineer at Interra Systems.

LAS VEGAS—Few technologies have arrived with as much impact as artificial intelligence/machine learning. The technology is disrupting established business models, changing whole categories of jobs and supercharging the pace of innovation. Still, the technology’s integration is nuanced.

“There has been a lot of hype around AI, but now we’re seeing very specific use cases that actually do give you a competitive advantage,” said Alfonso Peletier, CEO and founder of Epic Labs.

Jon Klein, president of Vilynx, agreed. “What opens people’s eyes is real results. For most media companies, seeing is believing.” Klein will present today’s session “AI Is NOT Technology. It’s Strategy,” in the AI+Cloud Campus.

The AI+Cloud Campus, which is on the show floor, is holding a number of 30-minute sessions through Thursday. Sponsors of the pavilion include AWS, Google Cloud, Sony, Harmonic and Vilynx.

Attendees at NAB Show will get to see firsthand just what machine learning can accomplish. Here’s a taste.

METADATA EXTRACTION, CONTENT RECOMMENDATION

As venture capitalist Mario Gavira and others have observed, Netflix derives much of its success from the clever exploitation of data. It knows what its users watch, when they watch it, how long they watch it for, on what devices and much more. They’ve leveraged this data to not only improve content delivery, but to tailor the creation of original content with the data-backed confidence that it will find a receptive audience. Broadcasters, production companies and other video businesses have more than enough data, and machine learning technologies are helping them level the playing field.

Large production companies have so much content that often languishes unused and unorganized because it’s too expensive to pay people to tag it, said Ryan Steelberg, president of Veritone. Thanks to the improvements in deep learning and computer vision, this video can now be ingested and tagged automatically.

The so-called metadata enrichment has grown very sophisticated using AI, Steelberg said. Veritone’s aiWARE “cognitive engine” can detect logos, objects, faces, spoken or written keywords and more and automatically generate keywords to make the video discoverable. The AI firm Vilynx is demonstrating its AI engine that can not only extract metadata but operates in tandem with a knowledge graph derived from scanning social media sites and over 50,000 websites to match content recommendations to trending topics.

The challenge to date in using AI-generated metadata tags is that the machine can struggle to parse what’s genuinely relevant from what’s extraneous, said Vilnyx Co-Founder and CTO Elisenda Bou-Balust. That’s why the company relied on an approach called unsupervised learning to train its algorithm. “We teach the machine skills,” Bou-Balust said. “It uses face detection to spot a face and if it doesn’t recognize who it is, it will search the internet to figure it out.” Extracting more precise metadata from video assets offers a kind of one-two punch for content firms, Steelberg said. First, it helps them understand what they have and, when it’s paired with customer insights, improves their ability to make personalized content recommendations to their viewers. “Media companies can harness this to make assumptions and correlations in real time about what’s working and what’s not,” he added.

INTELLIGENT CURATION & EDITING

“AI is super important for us,” said Andreas Jacobi, CEO and cofounder of Make. TV. The company’s Live Video Cloud uses a variety of machine vision and AI-based tools from several cloud services to improve the curation and qualification of live content or, as Jacobi puts it, to “separate the best from the mess.” Jacobi will do a deep dive on AI in his session “How AI Can Help Broadcasters Manage the Shifting Content Cosmos.”

“Companies are struggling to create content fast enough,” Jacobi said. Machine learning can help by not only flagging potentially promising video clips from streams of content but by automating the actual clipping, freeing editors to work on more in-depth and creative projects. “We definitely see AI aiding content producers on the editing side,” Jacobi said.

Evan Michaels, vice president of Video Product Management at Evolphin, agreed. At NAB Show, the company is demonstrating a machine learning-based tool that can prepopulate an editing timeline with relevant clips from ingested video. “It’s a rough cut of a highlight reel based on machine learning data,” Michaels explained. A human editor will still be on hand to make the final edit, but they’ll enjoy huge efficiency gains by not having to hunt for promising clips.

IMPROVED CONTENT DELIVERY

While improved compression codecs such as HEVC and VP9 and protocols like HLS and MPEG Dash have enabled over-the-top services to improve content delivery to viewers, there’s still room to wring greater efficiencies in the delivery pipeline. Epic Labs spent two years studying how machine learning could be applied to the challenge, said founder Alfonso Peletier. The result is LightFlow, which uses a combination of deep learning and computer vision to analyze video streams and aggregate analytics on end-point devices to create what Peletier dubs “content-aware optimization.”

Rather than simply compress video using a crude set of rules, LightFlow can “tweak the levers” of compression standards like H.264 and H.265 based on the kind of content that’s being transmitted and the kind of device that’s receiving it. The result, Peletier said, “is a much better user experience and a reduction of operating costs” associated with encoding, storage and CDN usage.

YOUR ROBOT ASSISTANT

One unifying theme among those working on machine learning is how the wave of video data (professional and user-generated) and viewer analytics has barely crested. “We’re just about to be flooded with a ton of new data — everyone is producing more content, and the sheer number of devices we have for distribution is huge,” Steelberg observed. In such an environment, it’s actually impossible to make sense of, manage and monetize this deluge without machines. But the reliance on AI tools won’t necessarily spell doom for those who make a living in the impacted industries.

“I don’t think machines will do creative things,” Bou-Balust said. “Machines will be assistants. They’ll be your super powers.”

LAS VEGAS—PBS has announced that it is working with Eluvio, a developer of software for managing and distributing large form content and GrayMeta, a provider of AI/ML technology, on a proof of concept for a dedicated blockchain environment that could allow PBS member stations to optimize their collective connective resources and utilize machine learning, metadata and microservices to create more efficient workflows and services. PBS made the announcement at the 2019 TechCon in Las Vegas.

“We’re working to create a blockchain fabric that stations would be able to opt into and utilize it in a dedicated environment within the PBS member station community in order to efficiently use various services,” said Renard Jenkins, vice president PBS Operations, Engineering & Distribution, adding that such services could include transcoding, datamining and excess resource sharing, for example.

Jenkins said PBS has been working with Eluvio and GrayMeta on the proof of concept for the last six months and that Eluvio CEO Michelle Munson presented various ideas and possible use cases during her keynote at PBS TechCon.

“We showed station representatives what we are working to architect and present to them for their input so we can move forward together,” he said.

At the conference, PBS officials updated attendees on its efforts to develop a new media supply chain that uses advances in cloud, networking and datamining to help them better manage the creation and distribution of content to multiple platforms. Jenkins and his team have spent three years planning and developing new workflows that will take advantage of the cloud as well as the move towards virtualization, and microservices enabled by SaaS (software as a service).

PBS’s relationship with its member stations is different than the relationship between a network and its affiliates or a commercial station group and its member stations. While a commercial station group may adopt a common platform from a single vendor for all of its stations, PBS’s stations are more like a federated structure, with each station its own entity with its own solutions.

“Even though we have different systems and different problems that we’re trying to solve, we are all essentially in the same boat in regard to what we have to do,” Jenkins said. “But what we do, we’re all marching along the same road together.”

Jenkins and his team examined every element along the content creation process from acquisition, editing, QC and transcoding to delivery and broke it into three phases:

Media Gateway, a cloud-based single point of entry for all media that also allows for other processes such as transcoding and compliance within the cloud. Media Bus, which is the “orchestrator” that controls and manipulates the movement of content; and Media Distribution, which would cover linear playout, OTT, B2B and B2C delivery out of a single plant.

PBS reviewed 339 proposals for the media gateway and media distribution segments and will announce the final vendors at the NAB Show, according to Jenkins. “We’re still in the design phase for media orchestration,” which he said will probably be announced in the fall. Jenkins expects to launch the gateway “within the next two to three months.”

MODERN DAY MEDIA COMPANY

Duane Smith and his staff at Oregon Public Broadcasting have also been working on new digital workflows for the past three years. Their goal, according to Smith, who is the vice president of technology for OPB, was to transform their network from a broadcast-centric facility to a facility that addresses the new multichannel, multiplatform world of today.

The planning was “part of a larger strategy about, how do we organize ourselves to be a modern day media company and not just a broadcast company, because a lot of our limitations derived from the fact that our entire infrastructure grew up around the broadcast world, and while that’s really good for over the air, none of those systems play in today’s media consumption landscape,” Smith said.

Smith’s task is to unite OPB’s nine production units around a standardized workflow flexible enough to handle the increasing myriad of tasks in the content creation process.

“The heart of our system is the media asset management system,” Smith said. “All the content is going to be registered there, all of the editing, all of the metadata is going to be added as required. And for all of those distribution points, we figured out how to abstract them down to a ‘checkbox.’”

The goal, according to Smith, is to create a workflow agile enough so that when new content creation methods are added, they don’t have a negative impact. “We didn’t want to have to disrupt our internal workflows to get content to a new platform,” he said.

The project consists of five “milestones,” according to Smith. “We’re about a month away from the end of milestone one, which is all the integration; the asset management system is being integrated with our CMS, our television automation and radio automation,” he said. “I think there’s 17 integrations we’re dealing with right now.

“Every aspect of content creation in our organization is going to be completely different,” Smith added. “There’s not one person or one process that is not being adjusted and standardized in a way that streamlines the process because a lot of our workflows and processes are left over from the days of analog.”

[Read: PBS Adopts Spec To Streamline HD Delivery]

Smith expects their new system will go live in phases starting in August and hopefully wrapped up by the end of September.

PREPARING FOR NEXT GEN TV

At Arizona PBS, CTO Ian MacSpadden has been working with the Pearl TV Group in the Phoenix Model Market project to test ATSC 3.0. As a member of PBS’s ATSC 3.0 engineering advisory committee, he and other engineering directors and CTOs from PBS are learning how to best educate broadcasters on the potential benefits of the Next Gen TV format as well as recommend best practices on how to implement it.

The promise of 4K/UHD over the air via ATSC 3.0 is one of the most attractive benefits of the standard. MacSpadden and his staff at the Phoenix-based PBS station were among the first stations in the area to shoot in 4K for their local programming. This gave the Pearl TV Group a valuable trove of high res content.

“We were one of the first stations in the market shooting and mastering, experimenting in 4K production, mastering in 1080p with HDR,” MacSpadden said. “So the Pearl Group was very interested because all of the rest of their participants were news stations and none of them are shooting in 4K.”

Handling such data-intense files though presents its own set of challenges, according to MacSpadden.

“What we’ve learned is that even though we’re shooting and mastering in 4K sometimes just getting that content to a station for air can be a challenge,” he said. “Our editors were trying to figure out how to do an HDR downmix and save it off into a file from their edit platform in a manner that the folks at the Pearl TV stations could read. We eventually found a commonality in that they were using the AJA KiPro Ultra 4K record and playback device.

“We’re still trying to work on tweaking a file to do a direct export and this will be a challenge for whatever platform stations use,” MacSpadden continued. “These are all things we are learning along the way as we figure out how to create ATSC 3.0 content through the entire production chain.”

Developing new content for devices beyond the living room TV set has prompted MacSpadden and his staff to more carefully plan ahead before production.

“This year, we’re actually trying to say, ‘OK, we’re also targeting the digital end when looking at the components of when we go out to shoot these shows; how do we put together digital sidecar content that will be these minute and a half to four minute exlusives that will be available on the web, to subscribers, or other distribution chains we decide to have available,’” he said.

MacSpadden compared learning a new content creation workflow to a new exercise routine.

“You have to get the muscle memory in place,” he said. “I need to make sure I’m covering all my bases and exercise all the muscles that I need.”

Audio technology has changed faces many times during the decades I’ve been in the industry. However, the biggest change by far was the transition from analog to digital, and once audio was turned into bits and bytes it enabled us to do things that seemed inconceivable in the past. Not only do we no longer record linearly, most of us no longer do anything in a linear fashion, with random access now being the way we record, work, and live our lives.

We’re currently undergoing another monumental transition in the audio industry, when audio technology can enhance user experience and drive quality forward or increase individual isolation, with instantaneous delivery often done at the expense of quality. The following technologies are all linchpins in our audio future and how we use or abuse them will determine the course of professional broadcast audio production.

MACHINE LEARNING

Most discussions of artificial intelligence seem to focus on robots taking jobs from humans while also performing all of our mundane household chores, leaving us simultaneously unemployed and free to get in some really serious binge watching. AI already has the ability to analyze and learn from itself, a process called “machine learning,” and this has helped create some powerful tools for audio production.

It is a brilliant use of computing power to have processes analyze themselves and their outputs or, in our case, analyze what we’re doing and offer suggestions to improve how and what we do. Anyone who mixes using presets is already relying on someone else’s idea of a good starting place, but machine learning takes this concept to the next level by analyzing in real time rather than just giving us a static starting point.

I was among the skeptics when it came to what are often touted as “automatic mix” tools until I discovered they are very useful for analysis, similar to using console meters in spectrum mode during mixes to monitor the overall tonal balance. These tools can act as a second set of ears listening for you, and that is almost always a positive thing. These machine learning audio tools are perfect for handling mundane tasks such as compliance and correction, which means they may one day free us from managing loudness and monitoring immersive audio down mixes.

IMMERSIVE AUDIO TOOLS

Immersive audio is the most exciting development for mix engineers since 5.1 surround, and it is the technology that makes me want to get back into daily audio production work. It seems like every month there is a new surround microphone coming to market to help create immersive content.

The possibilities for creativity and error with immersive audio are immense. Fortunately, there are now more tools than ever to help keep things from getting out of hand. Multichannel meters and panners that work across immersive busses are being released in software as well as in hardware form. These tools are accessible to just about everyone now, to the point that even the two primary workstations in my home studio, Pro Tools and Nuendo, include immersive panning, bussing, monitoring and authoring tool connectivity.

VIRTUAL AND AUGMENTED REALITY

Before 4K and HDR, 3D was the next big thing. Now the next big thing is supposed to be virtual reality. Slip on a VR visor and closed ear headphones and you become part of another world, that is, if the visuals are engaging enough to draw you in and the soundscape is enveloping and convincing.

This technology may be impressive, but a more intriguing alternative is augmented reality, where an enhanced version of the real world is presented through the screens of our devices. AR may actually present more challenges for sound designers than VR because many sounds need to be organic while others, such as audible cues, often work better when they sound totally artificial.

Since AR is viewed through portable device screens instead of visors, soundscapes are presented in virtualized surround on earbuds of dubious quality or through tiny device speakers. The challenge for audio professionals is to somehow make the augmented audio experience truly immersive and keep the viewer engaged when they can easily peer beyond their screen back to the real world.

THE COST OF CONVENIENCE

The era of personalized audio began with the introduction of the Sony Walkman in 1979 and it may finally have reached critical mass. We now find ourselves at the point where there is more individualized media consumption taking place than at any other time in history, yet very little of that media is the highest quality available.

Delivering digitized content using limited bandwidth requires widespread bit reduction to the point that it can be almost impossible to find digitally uncompressed content at the consumer end of the chain.

We’re balancing on the precipice of a very perilous chasm between convenience and quality. It may soon be the case that the only way to experience true uncompressed audio is to attend an unreinforced acoustic performance; or perhaps we’ll just have to start listening to the people around us.

Now more than ever, the goal of every audio engineer should be to create the best sounding, highest-quality audio imaginable every time we have the opportunity. We must strive to make people crave quality audio every time they listen or they will stop caring. We must use the incredible tools we now have at our disposal to create truly exciting, immersive and engaging audio for every single project, no matter where or on what device it will be heard.

In closing, this is my final Inside Audio column and I wish to thank everyone who has taken the time to read the column since I started filling it with words back in 2013. A very special thank you to Tom Butts for granting me this opportunity and to Terry Scutt for transforming my cogitations into sentences fit for print. Each column was an incredible learning experience and I can’t wait to read what Dennis Baxter will be sharing in this space in the future. His insights come from the leading-edge work he is doing in the very areas our industry is moving toward.

Jay Yeary is an audio engineer who has had the good fortune to spend his entire career in broadcasting. He is a member of AES, SBE, SMPTE, and TAB. He can be contacted throughTV Technologymagazine or attransientaudiolabs.com.

Machine learning and deep learning-based solutions are making a significant impact on media QC thanks to the availability of large GPU computing power and datasets. Using these technologies, media companies can automatically verify if audiovisual content meets compliance requirements. In regions of the world where nudity, adult content, violence, prohibited objects, substance abuse, and strong language are outlawed, media companies can leverage these technologies to increase compliance reliability and streamline their QC workflows, saving both time and money.

This article will explain why the success of a learning-based system depends heavily on the quality and quantity of datasets used. While publicly available datasets are good for general development of learning-based systems, they are not adequate for the specific requirements of content compliance in the media industry. Significant efforts are needed to build well-annotated quality datasets for the specific requirements of content compliance. If the training dataset is not well designed, then it is easy for an object detector to confuse guns with cell phones, for example.

ADVANCEMENTS IN ML/DL TECHNOLOGY

Content compliance can be a rather intricate process that involves analyzing metadata gathered from a variety of fundamental tasks, such as detecting objects inside frames, recognizing actions over several frames, classifying scenery, detecting specific events in audio or video tracks, classifying videos into specific activities or themes, converting speech to text, and detecting and recognizing faces.

In a traditional machine learning system, the features extracted from images for content compliance purposes were made by humans. Recent advancements in ML and DL have automated this process. A huge breakthrough in deep learning occurred in 2012 when AlexNet was designed. AlexNet is a convolutional neural network trained on 1.2 million real world images from a dataset called ImageNet for classification purposes. Images are classified into 1000 different categories, five layers and 60 million parameters, making AlexNet one of the most intricate and low-error-rate networks.

After AlexNet there were several additional developments in between the years of 2012 and 2015. Faster R-CNN, a deep neural network for object detection tasks, is one network that was proposed. While AlexNet addresses image classification, Faster R-CNN is designed to resolve object detection problems; therefore, it is more complex since it involves locating the object inside an image. Faster R-CNN recommends possible regions in an image that might contain an object and checks whether the proposed regions contain an object among the list of supported categories or not. If they do, the network returns the bounding box of the region containing the object and the name of object.

There are two key parts involved with constructing an ML network for QC. First, a QC solutions provider has to train the network on datasets so that the network can start recognizing objects of interest (e.g., guns, alcohol, cigarettes, belly buttons, etc.). Transfer learning is a technique that can be useful when training a network. Transfer learning reuses a trained model as a starting point for training on another dataset. This aids in training a network quickly for new types of objects and with less number of examples. Second, the trained network is applied in the media QC environment to make predictions about the presence of these objects in media files.

A critical factor of success for deep learning has been the availability of huge well labeled datasets. If datasets are well labeled, they can outperform the accuracy of human visual recognition. In fact, the top-performing dataset model achieved in accuracy of 96 percent in 2017.

THREE WAYS TO APPLY ML/DL TO MEDIA QC

ML/DL can be used for a range of different quality and compliance purposes in media workflows. Aside from detecting objects, the technology is useful for recognizing activity in a video frame, onscreen visual text, audio events, and whether captions are aligned correctly. Let’s look at three key ways that operators can use these techniques to their advantage.

Identifying explicit content is one area where ML/DL technology can be useful in the media environment. Object detection, activity recognition, audio and visual cues can be utilized to determine if there is nudity or minimal covering, mild sexual situations, or explicit sexual situations. Additionally, activity recognition and object detection can be used to identify violence, including the presence of guns, killing, and car crashes. In some regions of the world, the presence of alcohol and smoking in video content is prohibited. Operators can use object detection ML technology to identify alcohol as well as cigarettes, cigars and other vaping devices. Activity recognition plays a role in detecting the actual physical act of smoking.

CONCLUSION

Accuracy is crucial when it comes to quality control for media operations. If an operator misses a video scene with violence or alcohol, they run the risk of not adhering to content compliance requirements. Over the years, learning-based systems have evolved to where datasets are richer and higher in quality, improving the automatic identification of content for compliance purposes. With the latest innovations in ML/DL technology, operators can significantly increase the efficiency and accuracy of their media workflows. 

Interra Systems’ software-based QC solution has been integrated with the latest advancements in ML and AI technology, allowing operators to deliver exceptional audio-video quality on every device and comply with all regional content guidelines.

Shailesh Kumar is Associate Director of Engineering at Interra Systems.

HOLLYWOOD--The 2018 SMPTE conference opened on Monday with a overview of the opportunities M&E companies will one day pursue as private transportation transitions from cars driven by individuals to a future where self-driving vehicles turn drivers into passengers looking to be entertained.

Richard Merchant, Managing Director in Monitor Deloitte’s Strategy practice, and leader of Deloitte’s Media & Entertainment efforts on the Future of Mobility, painted a picture of where this transportation transformation that the M&E industry cannot ignore.

TV Technology Contributing Editor Phil Kurz sat down with Merchant after his address to discuss these opportunities in depth.

TV Technology:You seem very bullish on the future of autonomous vehicles. What gives you that confidence?

Greg Merchant: Deloitte has been doing a great deal of research studying the future of mobility and really delving deep into each of the industries it’s going to impact.

So even if you are wrong on the timing, it’s hard to argue against many of these impacts coming into society.

When you really look at the likelihood of everything transforming and the accelerating delivery of technologies that are making all of this possible, it’s hard to argue that it’s not going to change the way transportation works and the way mobility works sooner and permanently.

TVT:Earlier today in his keynote, Doug Davis of Intel quoted a Strategy Analytics study the company commissioned. It valued the autonomous vehicle market at $7 trillion by 2050. What do your figures show? And what slice of that will be related to media?

GM: The estimate we’ve done predicts about $2 trillion of economic activity every year. By 2030, the ecosystem should actually be humming along. In that, media specifically was pegged at $16 billion of economic impact every year.

But as I said in today’s speech, I would consider that more of a floor than a ceiling because there are more than just consumption opportunities we are talking about with the in-transit experience.

There are also going to be entirely new forms of experiences that are going to be developed across video, social, music, advertising, live entertainment, education and productivity. So, when you think about how all of those are going to be constructed both technically and experientially in the car, $16 billion will be on the low end.

TVT:Many of the speakers here at SMPTE today have made the point that the younger generation is receptive to ride sharing –a sign that Americans’ love affair with their cars may be coming to an end. But that generation was growing up in a time when the economy was flat. Given today’s strong economy, improvements in their financial means and growing ability to own their own cars, do you think their willingness to ride share will wane?

GM: I think we may see slight wavers, but I think the trend is headed in a direction where it’s not going to reverse. Put it this way, do you take more Ubers now than you did two years ago?

Ride sharing is actually expanding into different groups more than it is contracting. The conversation you have more often than not with leading and trailing millennials is about the reasons why they should ever buy a car.

Perhaps as they move out to the distant suburbs they may find an incentive to buy their own cars. But as long as they live around urban centers, ride sharing is only going to continue to grow.

TVT:Tell me more about the revenue possibilities this change in mobility will create for M&E.

GM: I think advertising is going to be revolutionized. You’ll be able to see different kinds of campaigns structured and targeted and sustained over the course of your daily commute, or even over the course of weeks.

You are going to be able to use location-based technologies to make them increasingly sophisticated and effective.

Ad insertion technologies for the inside of an automobile that are AR-enabled will absolutely change the way you interact with and view the world around you.

TVT:You mentioned in your presentation that billboards might evolve dramatically.

GM: Right now, the top of the line is a billboard that switches. It may, at most, have 30 different advertisements on it.

In the future, you could simply have a blank billboard that projects a customized ad for every single person that drives past it because it will know you are coming, and it will be able to do that seamlessly through an AR environment through your windshield.

TVT:Why would you even need that if you have all of the displays in your autonomous car?

GM: That’s part of the discussion. Do you still want to keep billboards in some way, shape or form? Remember, people still walk, they’ll see it outside the car.

But some billboards next to a high-volume highway, you probably don’t even need. If you can project an ad onto the billboard, you can most likely project a billboard.

That’s one area. The other that gets opened up, particularly with 5G, is that you are going to start to see ad-supported transportation –so the route you take home may change if someone pays a little extra to run you past their grocery store.

The cost of your ride will be determined by how much advertising you are willing to absorb on the trip.

I think you will actually see Uber partner with someone else to manage the cost of the ride.

TVT:What are some of your thoughts about how TV broadcasters and others might capitalize on this through their production businesses?

GM: Because of the development phase of the technology, most of what we are going to be limited by is our imagination.

Said another way, not everything is where it is going to be, but there is such a tailwind behind this technology that it will get there.

When you think about the different form factors you can put into a car –with ATSC 3.0 for instance—you’ll be able to have this integrated advertising. You’ll be able to have interactive like you’ve never dreamed of.

You’ll be able to play a game with an acquaintance halfway around the world while you are coming home from work and they are going to work.

You’ll be able to serve so much data into these delivery platforms at any one time.

TVT:What about the demand for production stemming from this anticipated mobility play?

GM: I think that is going to be a huge challenge. There is all of this technology on one side. On the other is the need to tell creative stories. There’s a need to tell a compelling version of what happened with completely engaging content.

I think there will be growing pains. It probably won’t be the most elegant experience as we get started.

But there is going to be an evolution as there has been with pretty much every media we come out with as people learn how to communicate through that particular medium.

You know, we are talking about something that could be an immersive 360-degree environment. So, to tell a story in that while you’re moving is going to be a little more complicated than just a two-dimensional story being told through your mobile phone.

I think it makes sense to start working on that now, particularly the technology and how to structure stories that get told in those environments as you go from your kitchen to the car, to the train, to a city bike –the ability to keep a story going and tell it across all those different media is going to be important. And there is a big production component to that.

But the real reason I think it makes a lot of sense is historically you’ve tried to simulate entertainment experiences and bring them into the car as best we can. Now we are talking about something where you have a contained environment where you could develop something that is the new standard of what people are going to want in their very own living room.

This isn’t something where the destination is only going to be the car. The destination could be the living room, but we are starting in the car.

The 2018 SMPTE conference continues through Thursday, Oct. 25. For more information, visit www.smpte2018.org

As emerging technologies go, artificial intelligence (AI) has certainly taken its time in making its presence felt on the world.

Surprising as that may be, the term AI has actually been around for almost 70 years, having been first coined back in 1955 by computer scientist John McCarthy, a.k.a the ‘father of AI’, the same year Emmett Brown invented time travel in the movie “Back to the Future”.

Since then, AI has experienced a largely stop-start existence, principally due to sporadic funding and below-par technology. In truth, the term AI has (arguably) gained more notoriety for storylines of killer robots (and the occasional Wall-e) hell-bent on destroying mankind than for its practical use and business benefits. But that’s all changing.

THE FOURTH INDUSTRIAL REVOLUTION

Thanks to breakthroughs in computing power, the advent and availability of big data, cloud hosting/storage, highly sophisticated software, and complex algorithms, the potential of AI is now starting to be fulfilled—with the business world being the biggest benefactors.

The market has reacted at pace. In recent years, billions of dollars have been invested by many of—if not all—of the world’s leading organizations into AI technologies (and companies), each of them looking to utilize some form of AI technology to future proof and improve their businesses, and/or create a competitive advantage.

Various estimates suggest more than $46 billion will be spent on AI services by 2020 by businesses, a figure rising to above $51 billion 12-months later.

WHERE WE ARE

Depending on what you have seen, heard, or read (fictional or not), you may have different ideas as to what AI is and is capable of at this stage. To offer some clarity there are three simple levels to be aware of:

1. Weak AI
AI capable of demonstrating human intelligence to carry out specific tasks.
2. Strong AI
AI capable of showing self-awareness, the ability to think and make decisions for itself to the same level as a human being.
3. AI Super Intelligence
AI showing superior levels of intelligence to human beings and fully in control of its existence.

For now—and for the foreseeable future—only Weak AI is currently relevant, so it’s time to remove any images of a leathered up, sunglasses wearing Arnold Schwarzenegger.

IT’S ALREADY HERE

Examples of weak AI have widely been adopted by many different business and are in use today—you just might not realize it.

In fact, it’s a fairly safe bet that you’ve already unknowingly encountered some form of AI and machine learning technology before you started reading this article. Maybe even before you got out of bed.

Have you ever used Apple’s personal assistant, Siri? Or Google’s Home? Perhaps you’ve noticed how your emails can often now detect questions being asked of you and automatically provide you with a readymade short reply? How about your smart phone that seems to be able to predict sentences you’re about to type as you’re typing?

For those that like a bit of online shopping (such as Amazon) or video streaming (Netflix), have you ever wondered how those personalized recommendations are determined? What about Facebook and its ability to not only recognize there are people in the photo you’ve posted, but can sometimes even identify them too?

They’re all using some form of AI, with the intention of bettering their customer’s experience, enhancing their financial opportunities and—unseen externally—improve their efficiency in workplace.

AI FOR ALL

“We are at the cusp of a new revolution, one that will ultimately transform every organization, every industry, and every public service across the world,” commented Ralph Haupter, president, Microsoft Asia. “I believe 2018 is the year that this will start to become mainstream, to begin to impact many aspects of our lives in a truly ubiquitous and meaningful way.”

You may be forgiven for thinking the implementation of AI is something exclusively for the Fortune 500’s of this world... but you’d be wrong. Today, things are so advanced that there is unlimited growth potential with AI companies designed to support business like yours.

In essence, AI is something that can be packaged up and purchased on a monthly basis, like your broadband or phone. This model reduces significant costs associated with more bespoke in-house solutions.

It also means the complexity is significantly removed (the ‘it doesn’t matter how it works as long as it works’ approach), meaning you won’t necessarily need to go and employ a qualified and expensive computer scientist.

“You don’t need to be a mathematics genius or have a PhD in software engineering to make sense of AI for your business,” said Gartner analyst Whit Andrews. “You don’t have to make massive investments in infrastructure and personnel in order to start applying AI’s potentially transformative technologies.

“These technologies will transform the nature of work and the workplace itself,” he added. “Machines will be able to carry out more of the tasks done by humans, complement the work that humans do, and even perform some tasks that go beyond what humans can do. As a result, some occupations will decline, others will grow, and many more will change.”

COMPETITIVE ADVANTAGE

So, why should you and your business consider implementing AI? Could it be just a fad? Will it really benefit you?

The answers are wide, extremely varied, and will be largely unique to your own business. There may be some obvious areas of your business you’d like to improve upon—be it financially or operationally motivated.

The magnitude and inevitability of AI cannot be ignored, nor underestimated. Many AI experts and professionals have described the potential impact of AI on businesses as being equivalent to the invention an adoption of the personal computer and email.

Some suggest that AI can help boost revenues by around 20 percent, whilst others warn that any business currently not at least thinking about adopting some form of AI could already be two years behind a rival. Further delays could even result in their demise further down the road due to losing a competitive advantage.

GETTING STARTED

The first thing to remember is what the purpose of AI actually is and identifying how it can benefit your business. From speaking to various professionals in the field, it can be narrowed down to two core reasons: solving existing problems and discovering/identifying new opportunities.

At its core, AI is fueled by data, which can come in many different forms for many different uses. Emails, newsletters, subscriptions, views to your website, downloads, and sales are just some examples of where data can be collected.

On its own, that data may not seem useful. However, contained within, it could be the difference between success and failure or profit and loss. This is where AI comes into play, and where it would be able to—in many instances—do the job of a human.

A famous quote within AI circles comes from Michael Palmer of the Association of National Advertisers sums it up well: “Data is just like crude [oil]. It’s valuable, but, if unrefined, it cannot really be used. It has to be changed into gas, plastic, chemicals, etc. to create a valuable entity that drives profitable activity; data must be broken down and analyzed for it to have value.”

Today, the bulk of that potentially valuable data held in companies is still not being utilized. “If we look at the amount of data which is actually being analyzed today, only 20 percent of the data we have is searchable and being used productively,” said IBM chairman, president, and CEO Ginni Rometty during a discussion on the subject. “The other 80 percent is held inside companies, generally not being used.”

IMPACT ON JOBS

One of the more universal drivers for AI is centered around automation—a word often which spreads fear when discussing the risks to people’s professions. Areas like administrative work (such as data processing and data collection) are widely seen as function where AI will support—or even replace—humans.

“If your work is repetitive and not creative, you will be gone very soon,” was a stark warning Dr. Roman V. Yampolskiy, a professor in the department of computer engineering and computer science at the Speed School of Engineering, University of Louisville.

Bart Selman, a professor of computer science at Cornell University, was quoted as saying: “A lot of large companies have middle management jobs where people manage other people at a very low-level in terms of keeping track of things like vacations and sick days. Those jobs I believe are at risk. Jobs that involve a large routine component. If you’ve made the proper investment, you can develop an AI system that can take over a good fraction of those jobs. A lot of big companies have a lot of those positions and will be looking at it.”

Examples of automation and loss of jobs can be seen all over the world today. Some of the headline grabbing stories include that of a Japanese law firm, Fukoku Mutual Life, which invested more than $1.7m building an AI platform with IBM Watson, has used the technology to replace more than 30 staff members. The annual savings for the firm is predicted to be around $1m a year.

“The next wave of economic dislocation won’t come from overseas,” President Obama stated in 2017 during his farewell address. “It will come from the relentless pace of automation that makes many good middle-class jobs obsolete.”

SPOTTING AN OPPORTUNITY

Another major area for AI adoption is within sales. There are now examples of where AI, using natural language processing software and specific algorithms, is able to spot patterns (in data), to identify new opportunities and provide a level of analysis on in just a few seconds.

This can vary greatly. Examples include gaining greater knowledge on customer’s viewing and buying behavior—i.e, what they’re looking at and when they most likely buy (such as pay day).

Another might be identifying the best time to send bespoke marketing and promotional materials or newsletters to specific customers rather than en masse as part of a one-size-fits-all strategy. Another could be automatically identifying cold customers that haven’t transacted with you for a while and send them updates, reminders, or special offers to help get things over the line.

THE FUTURE OF AI

These are just a handful of examples on how and where AI is already proving significant benefits and opportunities for businesses all over the world.

It’s important to realize these are all current technologies, with this article designed to demonstrate and educate you as a businessperson, what is available today, rather the stargazing into the future.

The AI revolution is here and it will—if it doesn’t already—play a key role in the future of the way your business runs and perform. When you decide to get on board is up to you. I’ll leave you with this quote from Gartner: “As vendors exploit AI software capabilities within business suites, enterprise applications, infrastructure support services, and the customer experience, your organization will need new or updated strategies. Ready or not, AI is coming to you.”

BURLINGTON, MASS.—In technology circles worldwide, artificial intelligence is a dominant topic of reflection, research and increasingly, implementation. AI is delivering real value in several areas of the media production and distribution chain, but applications are limited presently. Other areas are showing tremendous potential and are now being explored. While it’s only a matter of time before the technology and our collective expertise overcome current limitations, it’s interesting to note where we are now and where we’re heading.

ENHANCING CONTENT DISCOVERY

The most obvious application of AI today is around automated metadata extraction or content “indexing.” The process of taking content, applying an algorithm to derive time-based metadata that is then registered in an asset management system is greatly enhancing content discoverability. For example, an algorithm that converts speech to text aligns each word in the text to markers within the content, making it possible to locate where a certain word or phrase is spoken.

It’s also possible to index content to find voice signatures that identify where a certain speaker is speaking. Now you have two layers of time-based metadata that allow fine-grained searching. You can keep adding layers or “strata” that enable even more detailed searching. For example, you may want to search for any time a particular public figure said certain words within a specific time frame. AI algorithms are so advanced they can do more than identify faces—they can infer the mood of each person at any given time. As you apply more AI algorithms to a content library, you add more strata of time-based metadata, enabling increasingly sophisticated searching, while automating a time-intensive task that’s difficult and prohibitively expensive to achieve with humans.

Perhaps even more valuable is the ability to discover valuable content long after it has been archived. Often, we don’t know in advance what content will be significant. AI can help uncover valuable assets that are hiding in plain sight within a content library. With enough metadata strata you can triangulate in to find content you may not even know you have.

Often confused with speech-to-text but operating differently, phonetic search has extraordinary promise. A time-based indexing algorithm can convert text into phonemes that take you right to specific locations within a piece of content. The end result is that you can conduct keyword searches on libraries never before transcribed. The ability to find clips/content phonetically works extremely well and is in products now on the market.

AUTOMATING QUALITY CONTROL

Another significant AI use case is automated quality control. AI-driven QC solutions can churn through a library of clips at one keystroke and analyze a broad range of quality parameters. It can show, for example, whether a program master that is targeted for French audiences actually has French language audio tracks, eliminating the need for someone to watch an entire show track by track to know that it’s the right version all the way through—a tremendous time and resource savings.

AI can also be used to ensure the accuracy of closed captions. Without AI, manual caption checks are required. Sometimes captions are wrong in the program master or aren’t in sync with the audio. AI quality control can confirm not only that captions are there but also that they’re correct. Some QC solutions can even make needed timing corrections.

An application at the nation’s leading over-the-top media services provider shows precisely how this type of QC is implemented. One of their brand differentiators is quality, so the company runs all content through a gauntlet of automated QC services, such as audio levels and visual clarity, identifying whether a host of criteria are met to ensure the highest quality standards.

A NEW MEDIA SUPPLY CHAIN

In addition to enabling better search and quality control capabilities, AI is also being used for operational business intelligence. In the pre-digital days, the media supply chain was extremely disconnected, with all steps of the process separated by physical media such as tapes. Today’s workflows are much more integrated and hold tremendous potential for operational efficiency.

But an understanding of the process is critical. To optimize your efficiency, you must track what people are doing and how they’re doing it in ways that inform the operational view of the media supply chain. With this kind of business intelligence, you can see where your pipeline is bogging down. Where are the choke points? Is it a lack of ingest capacity? Or if 30 percent of time in edit suites is taken up by rendering, it may be time to offload editing to a render farm.

Through an integrated AI-enhanced media production platform, it’s possible to measure all operations. How many people are using the system? How many are working on a specific project? Where are the technical bottle-necks and so on? For the most part, this kind of data has been impossible to collect and analyze—until now, media workflows have largely been a black box. But as media organizations seek to do more with less, more transparency into day-to-day operations is critical to garner operational metrics. AI can provide deep business intelligence to optimize production workflows.

AUDIENCE ANALYTICS

It’s clearly an advantage to have intelligence and analytics about your audience and their viewing habits to inform business decisions. Beyond the production stages in the supply chain, AI has perhaps even more potential for optimizing content distribution, providing insights about consumption patterns that are impossible to glean otherwise.

You can see the potential for news outlets and broadcasts. As news has become a 24/7 web and social phenomenon, news organizations and journalists need to know what’s trending and what news is breaking. Understanding events in the broader world context can influence how stories are assigned and prioritized. Today, specialized cloud-based service providers use algorithms to comb the web and synthesize trends, tracking information that may be relevant to a news topic from different information sources as they unfold. This data can be dynamically updated and available to journalists in a dashboard view, for example. AI that informs the creative process by analyzing downstream consumption patterns presents extremely powerful application possibilities.

MOVING FORWARD

The major challenge for more widespread AI adoption is not developing the algorithms, it’s how best to use and integrate them. As we add more and more AI capabilities, how do we harness the power they hold in a way that gives tangible business benefits? How do we make the user experience elegant in light of growing dimensions of data? These are questions that will be addressed as the technology progresses and implementations evolve.

In short, AI is an area that’s ripe for innovation. Business models are still immature as companies assess how best to productize AI so it makes business sense both for users and providers. In the end, the closer we to get to the craft of storytelling the more of a human endeavor it becomes. AI is a tool that assists with storytelling; it doesn’t replace it.

Tim Claman is chief technology officer/vice president of product management for Avid.

“We don’t even know what we’ve got in our archive…”

Comments like this are all too common in the industry as media organizations have woken up to find their organically grown, sprawling content archives missing a key bit of information: metadata.

Television broadcasters and other owners of large content libraries are facing this problem due to the sheer volume of media assets locked up on data tape, with incomplete or idiosyncratic information about exactly what has been stored. Without complete and accurate metadata, it’s difficult to make decisions about the worth of a given media asset, and that makes content libraries difficult to monetize.

How many times have you heard of a film restoration archivist (while looking for something else) “finding” a supposedly lost asset in the vaults, and thus making a restoration project more complete, unique, newsworthy, etc.? Many media companies know all too well that the ability to store data versus its value are often misaligned. Companies have long been looking for tools to help drive value from assets that were created and acquired at great expense but that have become “lost” with little hope of being found.

DATA-DRIVEN CONTENT

Long ago (in internet time), we saw the rise of “big data,” and watched it become “analytics,” which has become “deep learning” (a refinement of machine learning), which is itself part of the catchall term called “artificial intelligence.”

But it’s all a variation on the same thing: performing algorithmic queries on deliberately acquired data about customers, products and services in order to yield actionable intelligence.

The media distribution business has embraced analytics. Successful media and entertainment distributors had to develop decisionmaking capabilities allowing them to respond rapidly to constant change, while also integrating rights, digital supply chain, web and social media data. This has helped distributors gain powerful insights about both their media assets and customers, which further informs the types of programming they are willing to invest in.

This use of analytics has opened a window into a critical link between customer preferences, monetization and what is in your media archive. In a recent study by Nielsen of the subscription video-on-demand (SVOD) consumer viewing habits, what’s driving 80 percent of the time spent viewing these services is the back catalog of content acquired by streaming services from television networks and studios. “…Our research shows most of the viewing time is spent with catalog programming,” said Nielsen’s COO Steve Hasker. In short, the content in media archives is driving the majority of the new and rapidly growing ways consumers are viewing media.

METADATA CREATING MONETIZATION

Creating metadata has typically been a manual process, where an informed and knowledgeable person can “tag” (or assign metadata to) a media asset, and armed with an appropriate taxonomy, standardize the description of the asset so that search techniques can be used with confidence.

But this approach doesn’t scale. There aren’t enough qualified people, let alone enough infrastructure to let users watch and tag content interactively. Further, video and audio need to be watched and listened to in real time, and there are literally thousands of years’ worth of material which may be trash—or treasure. No one will know without evaluating the asset.

Every rights holder has had to migrate their data—their companies’ precious capital assets—from an older medium to a newer one, simply to preserve its ability to be read in the future. This is a labor-and time-intensive process, which doesn’t add any intrinsic value to the media assets, but has to be done regardless. What if you could migrate your data once, add value to it as part of the migration, and then never need to migrate to another tape format again?

If you could “automagically” add metadata to your content archive—through a combination of AI techniques that watch and listen to your library content and build up a user-referenceable database of people, places, things, even sentiments—you would have the ability to create a new type of programming, where archival material could serve as context for current-day narratives much more easily than such programming is created today.

CONTENT AND THE RISE OF MACHINE LEARNING

As multiple sources of data proliferate within an organization, new AI and machine learning techniques evolve to make cross-correlations visible to departments that previously did not have visibility into this data. This promotes companies’ adoption of a “platform” approach to metadata collection, instead of an application-specific approach, where on-set or production metadata might not be considered useful in the digital supply chain or distribution metadata.

Applications that can analyze and correlate media assets with sources of user data can turn these static media assets into “data capital”—an asset class which will continue generating revenue throughout the life of the asset, much like real estate continuously generates income for its owners. Like real estate, the owners of data capital will have to invest in maintenance in order to keep generating revenue from that asset class. The ability to rerun an updated machine learning algorithm against an existing media asset library, correlated with more recent user data such as social media feeds or sensor data from location-based entertainment, may tease out previously overlooked narratives, or themes that can then inform new uses for the content.

However, the sheer size of media assets can make copying data from a passive archive like a tape library less agile than required for a quick reaction to market or celebrity news. A “data lake,” or globally scalable storage fabric consisting of scale-out NAS and geo-scale object storage systems, allows multiple workloads such as archiving, disaster recovery and collaboration to be executed without requiring multiple silos for each.

Now imagine you can weave an AI appliance into that data lake, which can be trained on your own data capital without having to migrate your data somewhere else for that purpose! This architecture will allow AI algorithms to perform facial recognition, object recognition, audio transcription, and even translation on media assets, in order to identify and track features which are useful to advertisers, piracy watchdogs, organizations tasked with identifying manipulated imagery, and others.

The appearance of toolsets which have moved “up the stack” from raw computer science algorithms, and become software frameworks and applications that run on scalable clustered appliances without having to migrate vast amounts of data has ushered in a new era of revenue generation. This in turn allows content rights-holders to breathe new life into their existing pipeline of content creation, content management and content distribution.

Tom Burns is Field CTO for Media and Entertainment, Dell EMC. He can be reached on Twitter at@TVBurns.

Automation has been infused into innumerable elements of our daily lives. From production and assembly lines to broadcast facilities around the world, the transition to automated processes and workflows now have deep roots, and have forever changed the way we work, shop and entertain.

A common concern across all appliances of automation is the reduction, or outright elimination, of the human element. While the transition from manual to automated operations will undoubtedly remove human error in many cases, there are certainly more sensitive tasks where the argument for maintaining a manual workflow remains strong.

In the broadcast space, the transition to automated closed-captioning workflows is one topic that breeds intense discussion both for and against. However, the technology has advanced enough to instill confidence with broadcasters in many of today’s top DMAs, clearly representing the future of this important application.

EVOLUTION OF SPEECH-TO-TEXT

Speech recognition dates back to the 1950s, with modest first steps focused on digits and the most basic English words. With consumer services such as Siri and Alexa continuing to improve with each product generation, it’s clear that speed and accuracy in speech-to-text recognition has come a long way. So goes the same with automated captioning technology, which benefits today from the strengths of modern artificial intelligence.

[For an opposing view, read: Is It Live, Or Is It Automated Speech Recognition?]

While different mandates on closed-captioning in broadcast television exist around the world, the unifying purpose ensures that deaf and hearing-impaired viewers can fully understand and enjoy the shows they watch. Beyond the hearing impaired, statistics show that one in six viewers worldwide prefer to receive closed captions with their content.

Production and transmission of live, manual closed captioning has long been challenged by high costs, availability, varied latency, and inconsistent accuracy rates. And it’s true that the transition to more automated, software-defined captioning workflows introduced a new series of challenges.

For example, while automatic speech recognition removes the cost and staffing concerns of manual captioning, the performance of early-generation servers and processors demonstrated accuracy and latency problems. These issues were especially magnified for broadcasters that must now deliver accurate closed captioning across a multichannel, multilingual, multistandard and multiplatform media landscape.

These concerns are rapidly diminishing. The accuracy of speech-to-text conversion across multiple languages continues to improve with the emergence of powerful, deep neural network advances. In fact, accuracy across today’s strongest platforms has been raised to 90 percent or higher. The statistical algorithms associated with these advances, coupled with larger multilingual databases to mine, more effectively interpret—and accurately spell out—the speech coming through the air feed or mix-minus microphone.

Meanwhile, the faster and more powerful processing of computing engines within automated captioning technology has significantly reduced the latency to near real-time. This achievement is particularly impressive given that automated captions took between 30–60 seconds on many systems as recently as one or two generations ago.

Additionally, as closed-captioning software matures, emerging applications to eliminate crosstalk, improve speaker identification and ignore interruptions are improving the overall quality and experience for hearing impaired viewers.

MARCHING FORWARD

Many of the above improvements are related to recent breakthroughs in machine learning technology, which have enabled a deep neural network approach to voice recognition. Machine learning not only strengthens accuracy, it also provides value through detection of different languages and the different ways that people speak.

That intelligence as it relates to different dialects will provide an overall boost to accuracy in closed captioning. Consider a live news operation, where on-premise, automated captioning software now directly integrates with newsroom computer systems without the need for a network connection. This will now help broadcasters strengthen availability—no concerns about a network outage taking the system down—and take advantage of news scripts and rundowns to learn and validate the spelling of local names and terminology. Both of these points were once major and justified arguments against automated captioning.

Automated captioning also enables the applications to be achieved efficiently on a larger scale—costs are lowered due to the transition from human stenographers to computer automation. And as there is a need to caption a growing amount of content, there is an economy of scale that drives the cost down even further as broadcasters automate these processes.

As systems grow more reliable and broadcasters grow more comfortable with the technology, they will also find new efficiencies and opportunities along the way. For one, broadcasters that need to cut into a regularly scheduled program with breaking news or weather alerts will no longer be forced to find qualified (and expensive) live captioners on short notice.

Improvements in captioning technology have also been timely around emerging needs, including networks tasked with captioning large libraries of prerecorded content. As more systems move to software-defined platforms, the captioning workflow for prerecorded and/or long-form content has been greatly simplified. Post-production staff can essentially drag-and-drop video files into a file-based workflow that extracts the audio track for text conversion. These files can then be delivered in various lengths and formats for a TV broadcast, the web, mobile and other platforms.

And with multiplatform reach, broadcasters also have opportunities to caption live and on-demand streams, ensuring that hearing-impaired and multilingual audiences watching online are properly served as well. The future of this technology is very exciting, especially with the knowledge that we’re really just beginning to reap the fruits of this technology.

Ken Frommert is president of ENCO.

Dateline 2014—at the time, the “digital universe” was growing at a phenomenal 40 percent annually and expected to continue “on into the next decade.” At the time, that growth rate reflected conglomerate sets of data that not only included people and enterprise, but included the relatively new term “Internet of Things (IoT).”

To a broadcast engineer, the term IoT used to mean “inductive output tube”—an alternative to the klystron, and both referencing transmitting tubes used in high-power TV transmitters, the latter in analog television and the former a most cost-effective device, which emerged full strength during the ATSC transition.

The modern day IoT may have equally as broad an impact for society as it did for the digital TV broadcast marketplace. The Internet of Things has propelled storage demands and solutions (including the object store) into the next universe, aiding and changing the perspective and dimensions of “big data” forever.

COMPREHENDING THE ZETTABYTE ERA

When the IDC conducted its study in 2014, they predicted the volume of unstructured data created and copied all over the world would reach 44 zettabytes (1 zettabyte = 2 to the 70th power bytes), i.e., 44 trillion GB, annually, by 2020. By perspective, just a year before that 2014 IDC prognostication, the amount of data created and stored in 2013 sat at a mere 4.4 trillion GB per year. If correct, the amount of data growth is outpacing Moore’s Law, and will increase tenfold in six years.

Ironically, according to that IDC report, the amount of useful data (if tagged and analyzed) grew by a much lesser amount. In 2013, only 22 percent of the data accumulated in the digital universe was considered “useful”—that is, it was relevant because it was meaningfully tagged or categorized and was searchable and retrievable.

By the year 2020, the IDC prediction reported in April 2014 stated that only 37 percent of the data collected will be useful because of that same criteria.

USELESS DATA RETENTION

So why do we continue to store data that isn’t useful? The simple answer: “Because we can.”

Irrespective of how, where, when or why we create this mass of data, we find that most companies, enterprises or individuals collect and save literally everything because, fundamentally, there isn’t the time to sort, catalog or even physically hit the delete key once the data is collected. On the personal level, think of how many VHS tapes or compact discs or DVDs you still have in boxes or on shelves in the basement or the attic.

Putting those collections into today’s perspective, all those memories are essentially just another set of data. If we digitized all those analog VHS tapes into compressed ones and zeros, we’d still have enormous sets of data that would likely remain unmanageable, ignored and probably lost in the digital quagmire of never-never land.

At least while in a tape format there was a storage container (the wrapper), information about the content (the metadata) and an easy methodology to catalog the content by orderly arrangements on shelves, boxes or with a 3x5 card catalog or even a digital picture of the box.

EXPONENTIAL EXPANSION

Production companies, news organizations, broadcasters all generate enormous amounts of data. The volumes continue to expand exponentially and will likely end up in the “no-where’s-land” of the digital landfill. For today, this enterprise digital repository is now an ambiguous, unknown depot that might be one of many ubiquitous “clouds”—some on premises, some in that atomic number 26 mountain place, some in a public cloud, and a lot more of it ending up in privately managed datacenters scattered around the globe.

For how long and what purpose do organizations intend to keep that data? It’s relatively inexpensive to hide those bits in a cloud and nearly zero cost to keep it there—until you want to retrieve it. However, to get meaningful use out of those bits, you needed to catalog it. Otherwise, you must pull it all down from the cloud, store it again (locally) and then search through it to find something usable.

For an enterprise of any size, this takes labor—which costs money. And that’s a resource that doesn’t grow automatically, like the data you and your friends and their friends are generating every second of the day.

INTELLIGENT DATA STORAGE

When you consider the daily couple of billion pieces of data “about” you, your friends and their friends, too, you can see the storage challenges which entities like Facebook, Google, Amazon, and the other social media or shopping platforms have on their hands. The difference is these companies have figured out how to intelligently collect the data, identifying each piece using artificial intelligence algorithms that are, incidentally, developed either by their own organization or acquired by buying another company with that expertise. Across each social group, they will cross-relationship every piece of their data and then store it in one-to-many of their “private” clouds—which are liberally dispersed data centers interconnected by networks based upon volumetric accessibility per region.

Their data is never stored just once. Instead, it is replicated multiple times for accessibility, protection and resiliency. How each organization diagnostically and dynamically protects that information and makes it nearly instantly retrievable is their secret sauce.

Yet today, some of the concepts and principles which social media companies have developed for their own applications are now becoming available to individuals and organizations. The goal in these products is to start diminishing the “uselessness” of the data by applying intelligent metadata that can then utilize more conventional search engine approaches for cataloging and retrieving those assets. These new AI-based approaches now differentiate the future from the more traditional legacy media asset management methodologies.

STRUCTURING THE UNSTRUCTURED

What we’ve learned by collecting huge sets of information about known places around the world is now supporting machine learning techniques that create accurate metadata tagged not just to a single image, but to an entire generation of data sets grouped as objects. Such information may use the angle of a shadow which then identifies a time of day, which, when coupled to a geographic (GPS) location, gives more information about the season or the atmospheric conditions. People in images can now be related to their siblings or parents, based upon data sets generated from favorites or albums. Road signs, window lettering on buildings, and other distinguishing characteristics add to the databases about the actual surroundings where that image, and those of others, were collected. What was heretofore considered useless information is now branded and repurposed by machines which “look” for this data and then catalog it without any direct human intervention.

Using these new autonomous techniques, every time a new piece of content (still image, sound or video) enters a system equipped with these technologies, the system turns that previously “unstructured” data into “structured” data that is then cataloged not just as a single image, but as collections of data sets bound into a global storage platform.

These are the roots of where we’re headed as the future of storage becomes an indirect, unsuspecting model that makes potentially useless data valuable again, for all.

Karl Paulsen is CTO at Diversified (www.diversifiedus.com) and a SMPTE Fellow. Read more about this and other storage topics in his book “Moving Media Storage Technologies.” Contact Karl atkpaulsen@diversifiedus.com.

ALEXANDRIA, VA.—You knew it was coming, right? When you walk around with more computing power in your pocket than it took to launch a Saturn V rocket to the moon, you get the hint that computers are increasingly doing work that we either don’t like doing or never could do before.

For example, take logging raw video and creating data files to let news organizations search for just the right clip when we need it. Need a shot of a burning building on East Main from November? Bingo… the AI system logged it and made it available on the server.

AI is also the tool behind giving viewers a better experience when they visit a station’s web page.

[Read: AI And The Digital Transformation]

“As consumers become more driven to personalized experiences, news stations need to keep up with dynamic content,” said Drew Martin, technical product manager for Grass Valley. “AI can provide rich end-user experiences with minimal manpower. This generates more viewership and small operating costs.”

WORKFLOW BENEFITS

And AI is not just for the viewers.

“AI can bring huge potential benefits to broadcasters, with particular relevance in areas of the workflow that are labor-and time-intensive—like ingest,” Martin said. “By enabling broadcasters to track how operations are being used across their organization, AI-based solutions can create more efficient operations and bring costs down by identifying trends. As broadcasters of all sizes are under pressure to produce more with lower budgets, AI-based solutions can help them focus their resources on creating more compelling content.”

Although Grass Valley has no current AI-related products, Martin said the company plans to add AI functionality to upcoming product releases.

Brick Eksten, chief technology officer for playout & networking at Frisco, Texas-based Imagine Communications, recommends using AI to test supply chain management enhancements before integrating them into a facility’s workflow.

“In a machine learning/artificial intelligence solution, the system could learn enough about the content types [by watching content] and could experiment with various combinations in an offline environment, until you have sufficient confidence that it is providing better management of the supply chain in real time than manual methods, optimizing for cost and quality at the level of each individual piece of content,” he said.

Another potential role is in monitoring. “An AI-assisted multiviewer could provide more in-depth information about each signal, but also put that in context across all of the individual devices that make up that particular channel of content,” Eksten said. “Today we monitor by exception; Tomorrow that monitoring will be more predictive and seamless.”

HUMAN OVERSEER

As promising as AI technology is, many operations touched by AI still need to have a human overseer to ensure smooth operations.

“We’re already seeing artificial intelligence being used as a tool to create content like highlight clips, with Aspera being used for the ingest of video content and the automated delivery of the produced assets,” said Richard Heitmann, vice president of Aspera, an offshoot of IBM. “The natural next step to automated production is automatic publication of personalized media experiences. But we are still in the early days of the technology, and there is a human-review element that won’t go away anytime soon.”

IBM has been developing AI applications for decades, and Aspera has been leveraging that background for its broadcast products.

In April 2018, IBM partnered with the Masters to bring cognitive highlights to the golf tournament, according to Heitmann. “IBM’s AI technology quickly identified key highlights based on cheering, high fives, commentary and TV graphics such as banners within specific video frames,” he said. “As a result, video editors were able to use Aspera technology to distribute highlight reels at high speed in near-real-time for fans.”

INCREASINGLY IMPORTANT ROLE

For broadcasters, it’s all about what you do with your bandwidth—the more effectively a broadcaster uses its broadcast bandwidth, the more profitable it can be. For this reason, AI products can now address bandwidth, including learning from one encoding session to improve the next.

“AI plays an increasingly important role in video encoding, where it can significantly help improve workflows,” said Reinhard Grandl, director of product management for Austria-based Bitmovin. “By continuously learning the parameters used in previous encodes, AI-optimized settings can be applied to every new video file. Furthermore, every asset that will be encoded with our service helps to train this machine-learning model and makes the prediction for future encodings more accurate. This results in faster processing times and significantly higher quality with no increase in bandwidth.”

The savings from properly configured AI-driven encoding are substantial, Grandl said.

“Netflix, for instance, estimates that its use of AI to automate workflows and reduce customer churn saves the company around $1 billion annually,” he said. “This not only increases the quality of experience and quality of service for users, but also reduces the number of bits required to achieve the same quality stream. YouTube is also at the forefront of using AI to reduce overall video latency and encoding costs.”

Netflix and YouTube are noteworthy examples, but is there anything AI can do for a call-letter TV station?

“Call-letter stations can receive immediate benefits from TVU Networks’ products with AI,” said Paul Shen, CEO of Mountain View, Calif.-based TVU Networks. “For example, the TVU Transcriber service is available today and ensures FCC compliance of any video content a station puts on air, on social media or any digital media platform. The AI engine in Transcriber detects the need for closed captioning in content and will automatically transcribe the missing speech as closed captions. In addition, TVU Transcriber uses AI to detect profanity and can automatically mute the audio.”

The tantalizing ability to deliver targeted content to web viewers is also within reach of TV stations, Shen said.

[Read: Media 4.0: Using AI To Meet Viewers’ Preferences]

“With the TVU MediaMind Platform, all digital and broadcast production groups can truly collaborate to cover the same story, while allowing each group to customize and deliver the completed program based on viewer demographics,” he said. “As a result, a station can cost-effectively create targeted content and allow it to better serve digital and broadcast viewers using the same raw videos. This becomes a truly story-centric workflow.”

Not only will a properly configured AI system process video as it is ingested, it can also dip deep into existing library files and process those.

AI functionality has recently been integrated into Prime Focus Technology’s, Clear Media ERP, Media Asset Management system, according to T Shobhana, vice president and global head of marketing & communications for the company. “[Clear] helps automatically recognize elements within audio and video, and generate associated metadata, making it easier to sort, locate and use content across all MAM workflows,” she said. “With Clear, content owners no longer have to rely only on manual effort to tag and catalog assets, as this is a time-consuming and expensive process.”

However, keep in mind that AI capability in MAM systems is not completely hands-off from a human standpoint—yet.

“All these functionalities require human review and quality control right now, but one of the key characteristics of AI and machine learning is the ability to learn and improve over time,” Shobhana said, “so we expect these functions to continue to evolve going forward.”

Over the past five years, artificial intelligence has moved out of the laboratory and into real products—you only have to go as far as Apple’s Siri and Google’s Alexa to find examples in the real world. The idea of a computerized assistant has now become real to millions, and that increases the pressure for similar machine aids in various professions and industries… including broadcasting.

It’s clear that the most efficient use of bandwidth and the ability to quickly create targeted programming are of great interest to broadcasters, and artificial intelligence is helping to make that possible.

AMSTERDAM—At the 2018 IBC Show, live IP video solutions provider TVU Networks will focus on TVU MediaMind platform, which uses artificial intelligence to drive how media content can be produced and consumed.

“At IBC we will be demonstrating solutions that deliver real ROI benefits today, as well as pointing towards an AI-led future for broadcast acquisition, production and distribution,” said Paul Shen, CEO of TVU Networks. “We look forward to demonstrating at IBC how our customers can embrace the exciting AI-driven changes that are about to revolutionize the media industry.”

Visitors to TVU’s IBC booth can also check out its TVU One field transmitter, which supports HEVC and utilizes TVU’s patented Inverse IS+ (StatMux Plus) transmission algorithm to transmit HD quality video with half-second latency

Also at the booth: the TVU MLink cellular 3G/4G LTE, satellite and microwave live video uplink transmitter’ the TVU Router, a portable broadband Internet access point that can provide over 200Mbps of secure, reliable, high-speed wireless Internet connectivity anywhere; and the TVU Anywhere mobile app, which provides turbo-boosted connectivity through bonding Wifi and cellular, enabling broadcasters to work anywhere.

TVU will also demo TVU Producer, a cloud-based live event production system that minimizes the complexity and cost of covering live video events from any location and the TVU Remote Production System, a cost-efficient solution capable of synchronizing up to 6 HD SDI signals over the standard internet without any OB vans, ideal for second and third football league tournaments.

TVU will also feature a section of its booth for attendees to learn more about how to stream, distribute and monetize video through the company’s TVU Grid marketplace. TVU Grid enables broadcasters to acquire live video from different sources and seamlessly distribute to thousands of Grid-enabled locations, as exemplified by major US broadcast networks such as CNN, which uses TVU Grid to make live video accessible to select TV station partners.

TVU will be in Stand 2.B28. For more information or to register, visit www.ibc.org.

At the root of the recent attention given to artificial intelligence is what is known, at a global level, as the “digital transformation.” Although predominantly utilized in the context of business, digital transformation (DX) has broad reaching impacts to many areas not the least of which are the television media and entertainment industries. DX is reaching the public and business sectors, numerous organizational activities, business process management (BPM), social media, and institutions ranging from government through education.

This industry-wide digital transformation is fueled, in part, by the increased focus and capabilities of artificial intelligence (AI) and by the applications of machine learning (ML). DX, AI and ML are augmented services either in the cloud or occasionally on premises. AI touches applications available from resources including the Apple iPhone, Google AI, IBM Watson, and a growing set of others.

INTELLIGENT SERVICES

Multiple new services applicable to broadcast, news and sports are using AI as part of their content creation, recognition, assembly, and distribution engines. In the video industry, according to companies such as TVU Networks and Veritone, we are now experiencing a sea change in the way video content is searched, produced, distributed, and consumed. Cognitive computing is transforming the way we use and generate video. Video customization, a frequent output of AI, is enabling individuals to see continued augmentation in how video is consumed and where or how it is being distributed.

Functionally, workflows utilizing AI begin at the point files are ingested using smart content management features that extract metadata using prediction engines. The AI-based engines determine flow, subject matter, relevance, plus other attributes which then generate relevant search components with high accuracy. AI is at the top of the compute-centric food chain (Fig. 1).

Other applicable AI-based services include advertising verification and sponsorship efficiency to track and verify brand name mentions, logos and characterization. For news purposes, AI techniques will categorize stories, interviews, breaking news and features – at both the local and the national level. For sports, player recognition and accumulated play or scoring data is used to self-generate melds of the game or statistics with better relevance than humans can – and do it in real time.

BEYOND SIMPLE RECOGNITION

AI is not just about facial recognition or venue classification or text/speech interpretation. AI utilizes machine learning, but it is not data mining. For media applications, AI is a key supporting agent in search-engines which engage sophisticated machine language-based algorithms to, for example, catalog images and sound for applications of metadata extraction or collection. Reducing the amount of manual human interaction needed to sort or tag images and sound is both supplementing and adding new value to archives and catalog platforms – and AI now allows those applications to go much further.

Indexing – previously a manual post ingest task - can now begin the instant that the video ingest and production processes start. Based on derived metadata, indexing allows real-time search to be built immediately using AI. And that information can be instantly shared (permissions pending) with others including users and other AI-based databases and libraries.

BASED IN THE CLOUD

Many of these new indexing platforms are built entirely on a cloud-based model. Using voice and object recognition, both live and pre-produced video clips can be indexed right down to the exact frame. Where once the sophistication of automated indexing amounted to scene change detection alone, today intelligent resource supplements can use information collected from other analysis to ascertain, e.g., people in the frame, voice or action recognition of non-visible speaking humans, if the objects in the scene are animals, building, automobiles, etc., and where the scene was shot based upon databases and interpretations from other images. All this at a reliability in the 80-85 percentile on a first pass; and even better accuracy on future passes.

AI allows services to build libraries of information that “learn” from previous identifications which in turn improves accuracy and speeds up the indexing and cataloging time with each task. The more the systems see, collect and validate the content, the better and faster the solutions get. Fig. 2 identifies the more common applications of AI for US companies in 2016 and those applications are expanding rapidly.

THE APPEAL FOR BROADCASTERS

AI for broadcasting exploits the efficiency of employing machines that can interpret and understand audience demands by using data management and filtering techniques poised to analyze content for specific themes and then create original content applicable to the individuals, locations and interaction of those people and placed in the images. These applications are particularly useful for taking raw/live content and boiling it down to rough cuts that can be stitched together for rapid release to OTT or mobile devices.

AI concepts literally “open the floodgates for how programs are produced and distributed,” according to Paul Shen, CEO of TVU Networks. Removing heretofore “human-delegated” barriers from the production process, digital and broadcast programming departments can create a single centralized search engine for raw materials such as live or recorded feeds, across all channels.

Furthermore, the integration of AI helps media companies better target specific audiences with more appropriate programs and advertisements – not unlike what we’re experiencing with social media such as at Facebook and Twitter. These entities are all utilizing varying degrees of AI and ML.

CONNECTED STRATEGY

Digital transformation aides in creating and optimizing new capabilities by leveraging the possibilities and opportunities of new and emerging technologies. However, the DX journey needs a staged approach defined with a clear roadmap. Stakeholders need to envision a connected world that is beyond silos, with a strategy that tears down internal vs. external constraints and appraises end goals that will continue to move as DX becomes the “de facto end-point” position going forward.

The IoT is also helping drive this connected strategy concept whereby systems previously confined to developing high level designs, architectures, and plans are now shifting to media and content. The capability to fine tune operational activities for business or manufacturing are now being applied to everyday consumer products such as smart homes, intrusion security and autonomous vehicles. With the explosion of content being generated and an overall demand to see more, faster and better; AI must be applied to broadcast, media and entertainment in order to satisfy that thirst.

BEWARE OF HYPE

Unfamiliar terms bring new “marketing” opportunities filled with anxiety that can yield to confusion. DX, one of the latest buzz words, is no different. As with any emerging and/or disruptive technology, there are tendencies to look or select those tech companies’ who offer “sexy” products or claim to answer ‘all your needs’ in a single offering. Just be weary, because, generally speaking, digital transformation should be considered “industry-agnostic,” and it is likely to encompass many offerings in multiple scenarios.

DX should start with business goals, identification of challenges, an exploration of current and future customers or needs, and then apply those findings to the context of the organization.

Digital transformation usually happens at different speeds. DX creates new partnerships which mutually leverage their collective synergies to produce a sum value which is greater than their individual parts. DX can merge disruptive entities (technologies and organizations) into harmonious entities. But beware, simply selecting a single offering without understanding and anticipating the overall impact to the organization (and its partners) can be detrimental to the success of the DX challenge. Some providers have indeed been “disruptive” in the sense of forcing bigger players to adapt or decease – that is only part of the agenda.

Potential adopters of AI-based solutions can learn from the new start-ups as well as those technology success stories we then hear about. We are only beginning to see the depth and interaction which AI and ML can bring to workflows and operations like media asset management equipped with automated content recognition or content assembly.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at kpaulsen@diversifiedus.com.

One of the more intriguing storylines of this year’s NAB Show was the realization that the biggest impact that automated intelligence will have on the television industry is that AI will allow production to change its focus from a program-centric process to a story-centric process where content is automatically produced, targeted and distributed to the viewer. In short, using AI will allow broadcasters to identify and tailor video content to individual tastes, and moreso, allow content producers to better monetize their assets.

This concept—known as “Media 4.0” where AI is used to automate the production and distribution of media to any device on-demand—has been around for several years. Whereas Media 1.0-3.0 represented the evolution of media from film and theater to radio and television broadcasting and finally to the current IP-based landscape, 4.0 can best be illustrated by the music industry’s use of big data and the cloud to provide the personalization of media consumption down to a granular level, according to Paul Shen, founder of TVU Networks, a provider of IP-based media production and distribution tools. Using Spotify as an example, Shen noted that the music streaming services’ popularity is not necessarily because of the depth of its library, but by the ability of its sophisticated algorithms to learn consumer’s musical tastes, creating “personalized radio.”

“I have 90 days of music collections on iTunes and I don’t listen to it at all, I use Spotify,” Shen said. “The reason is easy. I don’t have to do the work. Spotify offers the content the way I want it. It uses artificial intelligence to create the playlist.”

TEN YEARS BEHIND

This use of AI has helped the music industry and IP-based video services like YouTube and Netflix to leapfrog 10 years ahead of broadcast, according to Shen, who believes that current production and distribution methods are bogged down by the human-driven assembly-like process that can be changed to an automated process by using AI and machine learning with the unlocked power of metadata. To overcome these hurdles, TVU introduced its MediaMind cloud platform at the 2018 NAB Show.

MediaMind puts the concept of Media 4.0 into practice by using AI, and object and speech recognition to maximize the use of media assets, particularly in live production. This “smart media factory” works within the existing video production workflow but enhances it by automating the ability to identify video content and push it to a variety of platforms.

Shen uses President Trump’s recent North Korean prisoner release ceremony as an example of how the MediaMind concept could be put into practice.

“There were a lot of cameras there that were recording for hours, waiting for the moment [prisoners debarked from the plane], but they didn’t know exactly when,” Shen said. Using AI-powered facial recognition technology enabled by MediaMind, the cameras and mics could be automated to just focus on particular shots of interest to individual viewers, for example.

This CAS (Contribution Automation Solution) is one part of TVU’s MediaMind concept, but another equally important concept is the automation of the entire planning, acquisition and routing process. It integrates with the news system, router, camera, transmission devices and alert system. Once a story is created, the system will help manage resources by sending an alert message to the corresponding reporters and crews as well as all of the connected devices, such as cameras, transmission devices, routers and recording devices. Once content arrives into MediaMind, it becomes instantly searchable via metadata that is tagged to the content from acquisition onward. This advanced search capability gives MediaMind the ability to push content to the producer, rather than the producer having to manually access it.

“A producer says ‘I want to write a story about the return of the three Americans from North Korea,’” Shen notes as an example,” the content will appear next to him rather than him having to go search for it.”

IN PRACTICE

The MediaMind API is already being integrated into the workflows of 4-5 of TVU’s approximately 2,500 TV station customers. Shen says customers can pick and choose which parts of MediaMind they want to integrate into their processes, which covers the production chain from acquisition, editing, processing and distribution.

In today’s live televised production environment, nearly 99 percent of the raw material goes unused and therefore, un-monetized, according to Shen. Current production processes are tedious and don’t meet the demands of today’s viewer, who is used to using social media to provide the type of feedback that allows content to automatically be “pushed” to them.

Shen thinks Media 4.0, illustrated by TVU’s MediaMind platform, represents the future of live production—an integration between production and consumption—where viewers’ tastes, rather than the producers’ whims, will drive a more “story-centric” workflow.

“Media 4.0 is going to fundamentally change what’s needed to make the TV industry meet the demand of the audiences,” Shen said. “That’s what this whole initiative is about.”

LOS ANGELES — What if viewers – not editors — determined the trajectory of a story? Filmmaker Armen Perian intends to find out with the May 18 release of his new short film, “The Angry River.”

While not a tech guru himself, Perian, along with tech partner Crossbeat New York, have created a film with the interactivity of a video game but without a game controller.

The idea came to Perian during a long editing session. “Someone said, ‘Man, I wish we could just edit this thing with our minds,’” he says. “It was totally off-the-cuff, but the idea stuck with me.”

His new short film, starring Jim Beaver ("Deadwood") and Brooke Smith ("Bates Motel") employs eye-tracking technology to determine what a viewer is watching – where the viewer’s eye lingers — and then edits itself into one of five possible storylines, each matched to the viewer’s interest.

While enamored with the idea of game theory, Perian “wanted to preserve a cinematic experience, even though you’re watching it on your computer,” he says.

Like any other film, a viewer watches the screen, but when the viewer pays attention to something on screen, that drives the action, and the movie changes based on what is being viewed. 

[Read: The Next Big Step For AI? Understanding Video]

Crossbeat New York developed the eye-gaze detection and machine learning that power the story’s algorithm. Perian directed the film, which was shot in Oregon over four days. Alex Hall edited the film into five distinct narrative tracks. A custom-built algorithm turns the five different perspectives into a story that looks and plays like a traditional movie.

Perian’s production company Pomegranate Films has seen interest in acquiring the technology from studios and tech companies.

“I resist the idea that this project puts me in the tech world. I’m still a filmmaker, a story person. I just wouldn’t be able to tell this story without the tech,” he says. “It’s a tool. Even though it’s this sexy provocative new thing, it’s use is to serve the story.” 

"The Angry River" will premiere online May 18. 

SAN MATEO, CA.—Broadcast professionals responding to a survey at last month’s NAB Show foresee increasing reliance on AI and machine learning as well as hybrid cloud storage and a falloff on their reliance on tape-based storage in their future.

The survey, based on in-person interviews of more than 300 people at the show, revealed that 78 percent of broadcast professionals plan to use a combination of on-premise and cloud-based storage, also known as hybrid storage, to speed up media management.

Eighty percent plan to use AI and ML technologies to enrich metadata, the survey found. Among users of tape storage, 51 percent indicated they plan to move away from tape media over time.

[Read: The Next Big Step For AI? Understanding Video]

Among post-production professionals interviewed, 73 percent revealed they are frustrated with how desired media is located and retrieved and identified the issue as their primary storage challenge. Fifty percent said media management is more time-consuming today than it was three years ago.

Eight in 10 are thinking about using AI and/or ML technology to assist in tagging media, which indicates healthy interest in storage offering embedded rich metadata tags and metadata-bases search tools, the survey said

The survey also revealed a significant upturn in those who expect to use hybrid storage when compared to three years ago –78 percent versus 16 percent in 2015. Exclusive use of cloud storage appears to be headed lower, however, as 9 percent said they would be cloud-only in three years while 17 percent are today, it found. When it came to disk- versus tape-based storage as a primary storage medium, the former was clearly the preference, with 53 percent favoring disk and 32 percent choosing tape.

Among tape users, 51 percent said they plan to move away from tape in the next three years, the survey said.

Cloudian sells a scalable storage platform that consolidates, manages and protects enterprise data.

LONDON--As guests arrive at the Royal Wedding on May 19, Sky News will be using machine learning technology provided by Amazon Web Services and two partners to name guests and provide additional background.

Fans will be able to access the ‘Royal Wedding: Who’s Who Live’ via the Sky News app or via skynews.com. The technology enabling this enhanced user experience and deliver this service at scale, is being provided by Amazon Web Services (AWS) and two AWS technology partners, GrayMeta and UI Centric. 

As guests make their way into St. George’s Chapel, Windsor, AWS will capture live video and send it to cloud-based AWS Elemental Media Services for multiscreen viewing optimization. An on-demand video asset, including catch-up functionality, will also be generated. In parallel, Sky is combining the GrayMeta data analysis platform with the Amazon Rekognition video and image analysis service for real-time identification of guests and tagging with related information.

[Read: AWS Elemental Launches Media Services]

Finally, Sky News is using the Amazon CloudFront content delivery network to unify the content for rapid distribution to viewers. UI Centric has designed and developed the front-end application and video player to enhance the experience and user interface accessed by Sky News viewers.

Keith Wymbs, chief marketing officer for AWS Elemental, the video division from Amazon providing the encoding and cloud services for the event, said that AWS has a long relationship with Sky, adding that the capability will give Sky some insight into the capabilities of machine learning.

“It’s really a way to explore what we can do in a more nimble environment where we don’t have to provide a traditional workflow where you’re bolting servers into a rack and dealing with all the related hardware,” he said. “This gives Sky an understanding for exactly how much demand is going to be there so they can do a lot more experimentation and be first to market with things that are exciting for the end user.”

Video content from the application will also be made available on demand after the event. 

LAS VEGAS — The 2018 NAB Show is closed and in the books; with the attendance figures about 10% less than 2017 at just over 93,000 making the annual trek to Las Vegas. First impressions seemed to confirm the reduced attendance, at least to those in the Central and North Halls, as traffic seemed less than in the past. For 2018, NAB shifted many of the mainstream players back to South Upper and South Lower. The South Halls generally seemed jammed with people strolling to find the major players, at least from the broadcast equipment and content creation standpoint.

[Read: A Brief History Of The NAB Show]

For 2018, standout trends centered on the many evolving cloud-based workflows (including content and asset management, playout, and processing); the emerging applications and solutions for Internet Protocol (IP) infrastructures–based on the new SMPTE ST 2110 standards; availability of UHD/4K components (from the absurdly inexpensive to the traditional expectations); the practices of creating workflows for High Dynamic Range (HDR) and associated applications which allow users to generate both SDR and HDR; and last but not least, the true arrival of AR/VR and AI/ML … our new set of two-letter acronyms relating to virtual-reality and artificial intelligence.

IP IS HERE AND A REALITY

Perhaps the most evident of all the new technologies is the SDI-game changer — that is, real time video networking using IP.

From the IP-transition perspective, this was the first NAB since the adoption of SMPTE ST 2110 standards for managed professional media networks. Dozens of manufacturers brought many new IP‑centric products to the show. At the IP Showcase alone, 50+ vendors showed interconnected IP related products interoperating according to those standards which were produced by the SMPTE over the previous 18-24 months. Countless other vendors also amplified “IP” — regardless of what they meant by it.

[Read: What SMPTE-2110 Means For Broadcasters]

For this industry, “IP” is the new buzzword — the new direction, much like “cloud,” was only few short years ago. Yet, IP takes on many different forms — from compressed video, to media workflows, to the carriage of information technology, and — of course — those hot new entries steeped in the production and transmission of real time, full bandwidth, full bit-rate uncompressed audio-video (and metadata) in a networking environment. It seemed that without a doubt, IP is destined as an eventual replacement for SDI. Yet there still remain many hurdles to cross for full IP-adoption to be complete.

A collected endeavor of the many manufacturers who contributed to this transition was shown in the IP Showcase, found at the rear of the Central Hall. This year’s exhibit doubled in size over the 2017 NAB Show, reaching nearly 3,000 square feet. It featured a fully functioning, all IP-based video production control room staffed by volunteers and others, including students from Toronto’s Ryerson University, where presentations on IP technologies and applications were streamed live over the NAB channel.

Sponsored by the NAB and composed of trade and standards representatives from organizations including AIMS, VSF, AMWA, IABM, EBU, SMPTE and more; the showcase once again demonstrated working examples of the new IP video standards plus integration of the NMOS interface specifications. This year the showcase exhibits were arranged in an educational environment, letting visitors see and learn about the advanced capabilities of IP for professional video.

Potential IP adopters saw how 53 manufacturers addressed software-defined networking (SDN) alongside new tools aimed at diagnostics and operational management for IP implementations. Records showed some 1,030 attendees were scanned into the system as visitors.

AR/VR & MORE

Rippling down from January’s Consumer Electronics Show, held in this same location, was the enormous prominence of VR/AR (virtual and augmented reality) and AI/ML (artificial intelligence and machine learning). Throughout the show there were sessions and evolving products that support the industry’s new needs to create, manage, and deliver content to these emerging platforms. These cross-platform technologies are opening new doors, ones that are creating immersive and interactive media across social media and transmedia.

One of those “new era” production modes is that of eSports — the transformation of gaming to a real time, arena based live event. eSports attracts inventive players and is in turn changing production techniques that may show promise for aspiring new venues. This new gaming-environment (already attracting more than 40 million fans) could add far reaching opportunities for existing and future stadiums and arenas, especially when those locations are when not hosting other major league sporting events. eSports combines gaming and live “reality” television for both OTA and OTT, and pushes them into social media in a real-time domain. Look for many new programming opportunities across all forms of mobile communications and in-home entertainment.

MORE MOBILE VIDEO TRAFFIC

The biggest booth presence at NAB was Amazon — does this say something about the oncoming change? Here are some thoughts to ponder:

Program content production, ranging from long-form to user-generated short form, continues to explode — driving the technologies forward and the costs to produce that content downward. How that content is going to be consumed was a central undertone at NAB. According to Facebook’s Daniel Danker, “Fifty percent of all internet traffic is now delivery to mobile devices” and is “expected to be up to 75% in five years.”

One out of every five videos is “live” streaming. In August 2017, Facebook introduced “Watch,” a new platform for shows on Facebook. Watch is now available on mobile, on desktop and laptop, and in Facebook TV apps. Shows are made up of episodes — live or recorded — and follow a theme or storyline. And this is not where the story ends.

This, and dozens of similar stories, may indeed help drive the growth of the internet upward and outward. The NAB Show clearly showed this transition moving faster than ever, and the change is at a global level — noting that 73% of the homes in Sweden have 100 Mbps internet full time. All the service providers see this only accelerating the full adoption of mobile video communications and technologies like ATSC 3.0.

UHD, HDR & HFR

Continued emphasis on UHD was also echoed by the adoption of high dynamic range (HDR) and high frame rate (HFR). Several companies exhibited various means and methods to accommodate both the higher resolution characteristics of UHDTV (aka “4K”) and wider color gamut (WCG) perceptual capabilities found in HDR. Adding dimension to newer prospects, camera manufacturers and production solution providers alike are now making strong inroads into HFR video and the ability to produce both HDR and SDR (standard dynamic range aka “plain HD video”) in a simultaneous workflow.

Producing HDR and SDR, in concert with one another, comes with steeper challenges than when broadcasters moved from SD to HD video, or from stereo to surround. The complexities and processes with doing both, so that meaningful and proper images can be delivered to the consumer, showed its challenges at NAB as potential creators and users sought to understand which comes first, the HDR or the SDR — creating a sort of “chicken and the egg” perspective.

ATSC 3.0

The next generation OTA broadcast standard is complete, driving several traditional broadcast companies to develop products aimed at the initial rollouts. ATSC 3.0 is a game-changing standard designed to deliver better video and audio quality, not only for over-the-air (OTA), but also over-the-top (OTT).

Technology wise, ATSC 3.0 is an IP-based transmission standard designed on a five-layer stack akin to the seven-layer OSI stack utilized in IP-networking. The model allows for easy technology replacement and substitution as new features or developmental advances are rolled out. And ATSC 3.0 may have far reaching capabilities.

Audio improvements for ATSC 3.0 will be remarkable. Dolby AC-4, the new “next generation audio” (NGA) format, will include three Audio Element Formats: channel-based (like we have today with mono, stereo and surround); object-based (which is for immersive audio mixes) and includes audio signals and positioning metadata for customized audio programming; and scene-based, a sort of soundfield snapshot from a high-order ambisonic (i.e., a full-sphere surround sound technique) source that positions audio above and below the listener.

WHAT MIGHT 2019 BRING?

Given the significant emphasis on software, virtualization, the cloud, and the evolving flavors of audio and video imaging — plus where the internet will really take us — it is too early to figure out where we’ll be at the 2019 show. One thing is for sure though: change will continue and the industry will surely adapt. Stay tuned to see what survives and what stays as reality.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at kpaulsen@diversifiedus.com.

LAS VEGAS--“Get Ready for Machine Learning and Artificial Intelligence” is a new half-day program that will be part of The Next-Generation Media Technologies Conference at this year’s NAB Show.

Taking place Tuesday, April 10 from 9 a.m. to noon, the program will present six panels examining the impact of machine and artificial intelligence on production, post, filmmaking, and digital content creation.

“We’re looking forward to hosting some of the industries’ leading creatives and technologists as they reveal how neural networks and other advancements are enabling production techniques previously thought impossible,” said Chris Brown, NAB executive vice president, Conventions and Business Operations.   

The six sessions—taking place in North Meeting Room N257 at the Las Vegas Convention Center—will explore whether machine intelligence can increase productivity, efficiencies and creativity in production planning, animation, visual effects, post-production and localization. Panelists will also delve into the capabilities of today’s neural network-based tools, and their potential to alter jobs, workflows and the nature of content itself.

[Read: NAB Show Previewing Future Of Content Production In New Session]

The program kicks off with “The Evolution of Content Production Aided by Machine Learning,” featuring Usman Shakeel, WW Technical Leader, Media and Entertainment at Amazon Web Services. Next, Jean Bolot, vice president of Research and Innovation for Technicolor will present “Optimizing Production With Neural Networks.” At 9:45, Norman Hollyn, editor/professor at The USC School of Cinematic Arts and Todd Burke, principal solutions engineer at Adobe Systems will explore “How Machine Intelligence Is Transforming Editorial.”

The second set of sessions begins at 10:40 with Rick Grandy, senior solutions architect for Nvidia, speaking about “New Frontiers in Animation and Computer Graphics,” followed with “From Dailies to Master: Machine Intelligence Comes to Video Workflows,” at 11 a.m., featuring Weyron Enriques, vice president of product development at Deluxe Technology, Barbara Ford Grant, senior vice president, Digital Production Services, HBO, and Adrian Graham, cloud solutions architect, M&E, at Google.

The program concludes at 11:40 a.m. with Jeff Kember, technical director, Office of the CTO, Google Cloud discussing “The Future of Content With Machine Intelligence,” and the impact these new algorithms and this increased compute power could have on the entertainment industry.

WASHINGTON—Change is coming all along the content chain, from production to viewing, and the 2018 NAB Show will look to detail some of those changes with a special, three-act session. Produced by Amazon Web Services, the combined session will feature three presentations that focus on cloud-based media workflow combined with advanced machine learning for the delivery of immersive viewing experiences.

The first act of the session is “Race on the Red Planet: Chasing Our Creative Future With Curiosity and Opportunity.” This presentation will detail the OTT video event from NASA Jet Propulsion Laboratory rovers on Mars and how they tackle the challenges—like scaling up cloud resources, personalized ad insertion, language translation and captioning, and highlight package creation—in real time.

[TV Tech Experts Preview the 2018 NAB Show]

Futurist Marco Tempest will lead act two of the session, as he will take a peek into the role video and machine learning technologies will play on Earth and in the journey to Mars, as well as insights into creating more immersive experiences. Tempest is a creative technologist and advisor to organizations like NASA JPL.

The final act of the session will be a panel discussion between media brand and scientific technology leaders who are using cloud media workflows and looking at machine learning and artificial intelligence to boost those processes. NAB did not provide specific names of who will be on the panel in the official announcement.

[NAB Show: Technology for Storytelling]

The show will be encoded and delivered by AWS. A live stream will be available at live.awsevents.com, while a video-on-demand version will be available after the event.

The three-act session will take place on Wednesday, April 11 at 11 a.m. on the main stage in the North Hall of the Las Vegas Convention Center.