Throughout the 20th century, knowledge has continually expanded, stemming from the evolution of eras such as the industrial revolution, the space program, the atomic-bomb and nuclear energy and, of course, computers. In some cases, it may appear to the masses that artificial intelligence is about as common as a latte or peanut-butter-and-jelly sandwich. Yet the initial developments of AI date at least as far back as the 1950s steadily gaining ground and acceptance through the 1970s.

It wasn’t until the late 1970s and early 1980s that computer science began to emerge from a data-driven industry using large “main-frame” computational systems into platforms for everyday uses at a personal level. While the Mac and early PCs (beginning in the 1980s) were game changers, they were certainly limited on compute power and not designed to “learn” or render complex tasks with modeling or predictive capabilities.

Computers of that time relied on programming based essentially on an “if/then” language structure with simplified core languages aimed at solving repetitive problems driven by human interactions and coordination.

Trial and Error
As sufficient human resources and computer solutions began to develop and create “expert systems” (circa 1990s–2000s) the computer world rapidly moved into a new era—one built around “knowledge” and driven by language models with basic abilities to train itself using repetitive and predictive models not unlike those that infants or toddlers use to hear, absorb, repeat, say and “tune” their mind and physical stature to communicate, to learn physical principles (such as walking), coordination and more.

If you’ve ever observed a two-year old learn how to maneuver around a slick pool edge surface, make their way to the steps into a pool and gradually ease themselves into the water in a fashion that uses mental observation (aka “memory senses”), trial and error, and repetitive steps (slow walking or crawling) while a parent also observes the child’s actions to mitigate accidents or errors—you’ll quickly see, figuratively, what occurs in a set of computers or servers built to use software to train itself over time to create in part what will become a large language model (LLM) by applying its programming repetitively that eventually finds a way (i.e., to model) the original programming into a formidable model that reaches the desired goal.

Without Explicit Programming
Machine learning is just that kind of process and is the basis of AI, whereby computers can learn without being explicitly programmed. This generalization of ML has classifications that are utilized to differing degrees as diagrammed in the figure on Machine Learning Tasks (Fig. 1). Fundamentally, machine learning involves feeding data into coding algorithms that can then be trained (through repetition with large sums of data) to identify patterns and make predictions, which then form new data sets from which to better predict the appropriate outcome or solution.

A variety of applications such as image and speech recognition, natural language processing and recommendation platforms make up a new library of systems. The value relationship gained in this training process can be developed by (T) defining a task; (P) establishing a performance figure or criteria (i.e., how well did it do or how far off were the results); and (E) finding a resulting data set, the experience, given the data set it was provided for such an analysis.

Machine learning is a continual process whereby trials create results that get closer and closer to the “right solution” through reinforcement. Computers/servers learn which data set is then classified as “more right” or “less right” (i.e., more or less correct); and then stores those results that then modify the learning algorithm(s) until the practice gets as close to “fully right” as possible without “overshooting” the answer and risking a data overload or a false (hallucinogenic) outcome.

One downfall in ML is that the system may go “too far” (i.e., it has too many iterations), which then generates an exaggerated or wrong output and produces a “false-positive” that gets further from the proper or needed solution. Then one questions, “just how far does the generative process go before it is stopped?” When the system gets to the “regression” point—a case where the outputs are continuous rather than individual or discrete—then a categorizing sensing algorithm would necessitate supervisory termination or a resteering of the operation to a more dimensioned (contained) level, which restricts the continued processing.

So, in addition to the learning algorithm, there are sets of management algorithms that must be applied throughout the learning process to mitigate these so called “hallucination” possibilities. Remember the toddler in the pool, this manager may be the parent in this case, the individual who stops the child from being hurt or risking a task (T) that could be catastrophic in nature.

ML can (and is) used in many everyday solutions including email filtering, telephone SPAM filtering, anomaly detection in financial institutions, social media facial recognition, customer data analysis (purchase history, demographics) and trends, price adjustments (such as with non-incognito searching), and emerging advanced solutions including self-driving cars and medical diagnostics. The “balancing” apparatus must weigh multiple solutions, alternatives and decision points, which in turn keep a runaway situation from occurring, resulting in an unnatural or impossible situation or solution.

Industry Challenges-Bias & Fairness
Besides the rapidly developing capabilities, there are as many challenges in this evolving AI industry as there are opportunities. Data Bias and Fairness (e.g., in social media) is highly dependent on the data it has available for training. Bias can obviously lean toward and potentially lend to discriminatory solutions.

Privacy protection as well as security breaches head the users into areas that result in illegal or illegitimate practices. Given the ease in spinning up huge data server systems in the cloud, the possibility of running tens of thousands of iterations on passwords or account numbers means the risk to the customer (as in credit card fraud) can grow exponentially to the number of credit card holders. Banks and credit services use very complex AI models to protect their customers.

This in turn opens the door to another level of AI—that is risk, fraud protection analysis and monitoring. It’s a huge cost to the credit card companies, but one that must be spent in order to protect their integrity.

Another concern is in automation and the potential for job displacement. It is inevitable that some people will be displaced by automated AI solutions. In turn, these new dimensions are opening up new job opportunities across new sectors of the workforce, such as data analysis, machine learning management, data visualization, cloud tool development, laboratory assistants or managers who test and hone these algorithms.

There continue to be many misconceptions related to these new words and their actions. Buzzwords such as "machine learning," "deep learning" and "artificial intelligence" have many people thinking these are all the same thing whenever they hear the phrase “AI.” Regulations are being developed internationally and within our own legislatures that directly relate the “AI word” to machine learning or vice versa.

Most of these early “rules” are done in a pacifistic way, likely because the legislative authors have little actual knowledge or background in this area. Obviously, we risk going in the negative direction by reacting improperly or too rapidly—but things will surely happen that will need to be corrected further downstream. Doing nothing can be as risky as doing too much.

Easily Defined and Managed
As for the media and entertainment industry, efforts are well underway to put dimension on the topics of AI, ML and such. As with any of the previous standards developed, user inputs and user requirements become the foundation for the path towards a standardization process. We start with definitions that are crafted to applications, then refine the definitions that reinforce repeatable and useful applications. Through generous feedback and group participation, committee efforts put brackets around the fragments of the structures to the point that the systems can be managed easily, effectively and consistently.

For example, industry might use AI or ML to assemble, test or refine structures, which will in turn allow others to learn the proper and appropriate uses of this relatively new technology. Guidance is important, as such, SMPTE, through the development of Engineering Report (ER) 1010:2023 “Artificial Intelligence and Media,” is moving forward on this development and has a usable structure from which newbies and experienced personnel can gain value from.

Even the news media has interest in its future in AI. In their book “Artificial Intelligence in News Media: Current Perceptions and Future Outlook,” Mathias-Felipe de-Lima-Santos and Wilson Ceron state that “news media has been greatly disrupted by the potential of technologically driven approaches in the creation, production and distribution of news products and services.” l

Retrieving important value from content libraries is a huge problem that Media & Entertainment (M&E) companies are facing today. Through the help of machine learning (ML) and artificial intelligence (AI), these companies can share their content and extract value in the form of workflow efficiencies, better content creation and even additional revenue. ML is allowing companies to access metadata associated with every frame of video, making their content searchable and meaningful. Whether applying ML in real life or less time sensitive workflow ML makes vast amounts of video searchable in an instant.

In this whitepaper, you will read about:

• What machine learning is capable of and why it matters to extracting valuable data 
• Recognizing the power of ML and AI on these company's databases
• How to get started and what to expect with these tools
• Read the white paper now!

The White Paper is available for download here. 

WASHINGTON—Like any other technology, artificial intelligence and machine learning for video production and distribution came about in an effort to build a better mousetrap. Any product that does more with less effort has an advantage over products that don’t make the leap to the latest technology—that’s why self-driving car technology is often in the news.

There is no self-driving product for the television industry… no editing device that will automatically assemble a program, no camera that will point and adjust itself perfectly without human intervention, and no transmitter or distribution chain that will perfectly adjust itself to changing conditions and signal anomalies.

In Post
Those things are coming, however. There is no way to say exactly when, but my bet is that we are closer today to amazing artificial intelligence technology in the television industry than we are to the DTV transition in 2009. (And that seems like just a couple years ago!)

Meanwhile, there are several companies pioneering artificial intelligence and machine learning products that target a range of television applications. One of those is Blackmagic Design, which has AI functions in its DaVinci Resolve editing software.

“DaVinci Resolve Studio’s Magic Mask uses the DaVinci Neural Engine to automatically create masks for an entire person, object or specific feature, such as face or arms,” said Shawn Carlson, product specialist for DaVinci Resolve at Blackmagic Design. “DaVinci Neural Engine functions in Magic Mask offers specific human feature recognition for difficult isolation needs, like hair with bangs and exposed skin on a bearded face.”

Removing an object in a video shot can be difficult, especially if there is a lot of movement. DaVinci Resolve Studio users can remove unwanted objects using a combination of Power Windows, tracking and the object removal plug-in. The DaVinci Neural Engine analyzes the shot using machine learning and AI, and determines how to remove the object from the scene. Users can adjust various settings until the object disappears.

Carlson said that the DaVinci Neural Engine is for visual elements only at this time, and does not have any role with audio.

Streaming Data
The transport of streaming data is another function that benefits from AI and machine learning. With so much data moving so quickly there is no way that human observers can watch it all and compensate as necessary—it’s the perfect job for artificial intelligence.

Zixi is one company that uses AI to monitor streaming data and provide alerts and adjustments in the event of signal degradation. “We use AI/ML in two areas: video transport and video content analysis,” said Andrew Broadstone, senior director of product management at Zixi. “Low-level protocol measurements, such as round-trip time, network congestion and retransmission rate, are used to determine link quality and to anticipate upcoming signal degradation.”

Much of what Zixi tests is the transport stream quality, but the company’s quality measurements also drill down into the video encoding to ensure image and sound quality are maintained.

“Certain kinds of content analysis are independent of codec,” Broadstone said. “We also use AI and ML to estimate the perceptual quality of live video [VMAF] without a reference, but for this the video must be H.264/AVC transport stream format. In general, our customers overwhelmingly use H.264 since it is the most compatible format across devices.”

Broadstone said that Zixi’s IDP product uses tens of measurements collected every few seconds across all participants in a video workflow to determine what the company calls its “health score.” This health score lets IDP predict signal path quality and degradation.

“Zixi Health Score is the output of multiple models trained using gradient boosting across the entire Zixi data set with many months of data,” he added. “Zixi Health Score therefore is not a simple set of rules. However, it is typical to see the Health Score drop significantly when there is a sudden change in packet round-trip time, or if raw packet loss steadily increases.”

The aim of Zixi IDP is to anticipate problems and alert operators to the root cause, Broadstone said.

Cloud-Ready Monitoring
Monitoring data streams is also at the forefront of Interra System’s ORION, a real-time software-based, cloud-ready content monitoring system that enables service providers to deliver clean video. 

ORION provides real-time monitoring of IP/SDI/SDIoIP-based infrastructures that looks at all aspects of video streams such as QoS, QoE, closed captions, ad-insertion verification, reporting and troubleshooting. 

According to Ramandeep Singh Sandhu, senior management staff member at Interra Systems, ORION performs monitoring functions on hundreds of services simultaneously from a single platform, providing an operator with a single point of visibility and access to information such as status, alerts, alarms, visible impairments, error reports and triggered captures.

What are the typical anomalies that trigger an alert in ORION? “A total signal loss condition will typically be preceded by continuity counter errors, substantial reduction in program bitrates, high network jitter and packet drops,” Sandhu said. “QoS/QoE scores computed by ORION will also show significant dips in such cases.”

Identifying On-Screen Objects
As you might imagine, web-streaming specialist Amazon Web Services uses AI/ML for a range of applications. One example gives customers flexibility and precision when identifying on-screen objects. For example, AWS’s Media2Cloud can use Amazon Rekognition AI to identify that the on-screen object is a dog or something else.

According to Alex Burkleaux and Evan Statton of AWS, Rekognition is constantly adding new names, objects, and other features it can detect. For example, if a customer requires identification of specific kinds of dogs, Rekognition’s Custom Labels can be programmed to differentiate among images of different kinds of dogs, such as “labrador,” “terrier” or “boxer.”

Of course, recognizing faces is a frequent job for AWS Rekognition, and the faces of many celebrities are already in the database. Rekognition Face Search provides a mechanism for detecting people who are not part of the Celebrity Detection data set.

Where does all this AI power come from? Is it more processor power or better programming?

“They go hand in hand,” Burkleaux and Statton said. “Processing power and data are required to build and train machine-learning models. When you’re using AWS AI Services such as Amazon Rekognition and Amazon Transcribe, this is part handled by the managed service provided by AWS.”

AI/ML products may seem almost magical at times today, but they are only going to get more capable over time. Eventually, we may get to a Siri- or Alexa-like interface where you can simply describe what you want and have the service do the heavy lifting.

For example, you might say something like, “Analyze these six video clips and identify if any of the cars in them are Chevrolets” or “Monitor this data stream and report any conditions that either exceed standard parameters or consistently get close to a fault situation.” Of course, you might have an additional conversation to ensure the AI assistant understands what you are asking—that’s understandable.

It’s hard to say if that is five or 10 years in the future. It might be shown at next year’s NAB Show.

You can be sure that change is coming, and that there will be more AI/ML in the future.

Over the years, the complexity of video preparation and delivery has increased dramatically. First, the industry witnessed the move from tape to file-based workflows, followed by the transition from analog to digital. New formats and standards have also emerged, adding to the complexity of video delivery. 

Aside from these technology transformations, consumer viewing habits are shifting. Today’s viewers prefer OTT media services, with 76% of U.S. households subscribing to OTT services compared with 62% for traditional pay-TV, according to the latest research from Parks Associates. As broadcasters deliver a higher volume of content to a wider range of screens and global audiences, additional errors are being introduced into the workflow, potentially affecting video and audio quality.

Recent advancements in automated media quality control and monitoring systems are helping broadcasters deliver error-free video and audio on every screen. In particular, innovations in machine learning and artificial intelligence are pushing media QC and monitoring to the next level, increasing the accuracy and consistency of certain media tasks, including content classification, content categorization, lip sync checks and more.

MEDIA QC AND MONITORING IS EVOLVING

In the early stages of media QC and monitoring, automated systems were limited to simple tasks, such as checking the correctness of audio/video technical parameters, including resolution, frame rate, bitrate, content structure and container parameters.

Since then, media QC and monitoring has evolved. Today, broadcasters can check for perceptual errors using computer vision and standard audio processing techniques. These checks include interlace artifacts, defective pixels, dropouts, visual text recognition, compression and ghosting artifacts, loudness and language detection. 

With the rise of ML and its success in completing tasks such as content classification and object detection, the scope of media QC and monitoring has expanded. Now broadcasters are using advanced ML techniques capable of semantically understanding content for the purpose of content moderation, content classification, indexing and description generation. Let’s look at a few of the specific media applications that can be optimized with ML and AI technologies.

SPEEDING UP CONTENT COMPLIANCE WITH ML 

Monitoring and altering content in order to conform to different rules and regulations is one application that can greatly benefit from ML. Broadcasters must comply with a wide range of rules and regulations, which can vary from one region to another. 

Traditionally, broadcasters have maintained a pool of human moderators to manually filter content for regulatory compliance. Under a typical manual workflow, content is passed through multiple stages of review. If a review fails at any stage, the content goes back for editing. Manual content QC and monitoring is expensive, time-consuming and inaccurate. With so many global and regional aspects of content moderation, it is almost impossible for humans to carry out the job with 100% accuracy.

By automating this process, broadcasters can eliminate the limitations of manual content moderation, including the inability for people to memorize a significant number of visual symbols and the possibility for human error. With an automated QC and monitoring workflow, broadcasters can more rapidly and accurately check content for the presence of brand names, hate symbols, alcohol, violence, celebrity faces, vulgar speech captions and religious symbols. 

When using an automated system powered by ML, computer vision techniques and computer algorithms, the benefits are even greater. ML-based systems can handle huge and multiple content classification check lists without any major performance limitations, driving broadcast workflow efficiencies. 

However, it’s important to note that while current ML solutions are sophisticated and may be combined to create broader applications, they lack the real-world knowledge and human experience needed to create valid and acceptable outcomes on their own. Human input is still required to confirm the validity of patterns and help machines refine the result. Such human interactions are likely to define ML uses in the media industry for the foreseeable future.

ENSURING SUPERIOR QUALITY CAPTIONS WITH ML

Checking for the presence and accuracy of captions is another application area where ML has proven to be very effective. ML can be used to automatically generate captions where they are not present in the content, check the alignment between captions and audio, and check the correctness of the captions against the spoken audio. In addition, ML simplifies the identification of speakers in audio, ensuring that the correct punctuations are placed in captions. 

Ultimately, with ML, broadcasters can expedite the caption creation and verification processes for both live and VOD content, while ensuring that when content is delivered in multiple video quality levels within OTT video streams, the captions maintain a high quality.

Over the last decade, automatic speech recognition engines have achieved extremely high accuracy, up to 85%, via ML. Still, automatic speech engines face several challenges, such as robustness issues in noisy environments, the ability to handle variable accents, problems when multiple speakers are talking at the same time, and difficulty with kids’ voices (due to a lack of data to train ML models).

Keeping humans in the loop is imperative to resolve these challenges. By combining cutting-edge ML and automatic speech recognition technology with a manual review process, broadcasters can bring increased simplicity and cost savings to the creation, management and delivery of captions for traditional TV and video streaming.

ELIMINATING AV LIP SYNC ISSUES WITH ML

Synchronization between audio and video is a common issue today. Leveraging image processing and ML technology and deep neural networks, broadcasters can automatically detect audio and video sync errors. ML offers a faster and more precise approach to detecting audio lead and lag issues in media content, compared with the traditional approach of manually checking for lip sync errors. This allows broadcasters to provide a high quality of experience to viewers (QoE).

Through the power of ML, broadcasters can perform facial detection, facial tracking, lip detection, lip activity detection and speech identification. With an ML-based lip sync solution, typically one module uses video to extract faces and track lip movement. A second module  uses audio to extract audio features and a third ML module matches the movements with the audio features. Using this technique, it is possible to detect even one frame of synchronization issues.

CONCLUSION

The amount of content that broadcasters are delivering across the globe is massive. Ensuring a high-quality video experience on every screen is essential if broadcasters want to keep viewers satisfied. With automated QC and monitoring solutions featuring ML and AI technology, broadcasters are better placed to quickly and more accurately comply with industry and government regulations, deliver high-quality captions, classify and categorize content and eliminate lip sync issues. 

Anupama Anantharaman is vice president, Product Management, at Interra Systems.

EXTON, Pa.—IBM is joining the SCTE-ISBE Explorer Initiative Artificial Intelligence and Machine Learning workgroup, becoming the first member that is not part of the cable telecommunications industry, according to SCTE-ISBE.

The Explorer Initiative was founded in March of this year as an expansion of SCTE-ISBE’s standards program. The initiative brought leaders into seven new working groups to develop standards for AI and ML, smart cities, extended spectrum and more. Explorer working groups were selected for their potential to impact telecommunications infrastructure, take advantage of the benefits of cable’s 10G platform and improve society’s ability to cope with natural disasters and health crises, like COVID-19, says SCTE-ISBE.

IBM, as the new member, will collaborate with subject matter experts from across industries to develop AI and ML standards and best practices.

“Integrating advancements in AI and machine learning with the deployment of agile, open and secure software-defined networks will help usher in new innovations, many of which will transform the way we connect,” said Steve Canepa, global industry managing director, telecommunications, media and entertainment for IBM.

For more information, visit www.scte.org.  

OTTAWA—Trend alert: Artificial intelligence/machine learning (AI/ML) is becoming an integral part of the total TV production/playout process.

“AI/ML is shifting to provide tremendous value to broadcasters and content producers,” said Amro Shihadah, IdenTV’s Co-Founder & Chief Operating Officer for IdenTV, a McLean, Va.-based real time video analysis market researcher. “AI/ML is achieving this by transforming big data from a cost center and opaque set of structured/unstructured datasets into real-time actionable analytics and tools for big data search and recall, creating a better user experience, and generating revenue from new content distribution channels.”

Broadcast consultant Gary Olson, who has just released the second version of his book, “Planning and Designing the IP Broadcast Facility—A New Puzzle To Solve,’’ says the technology is already showing up in elements of the production chain and is expected to expand its footprint.

“I see AI/ML appearing in editing, graphics and media management products in 2020,” Olson said. As the year progresses, “it will be interesting to see which vendors will claim their products have AI or ML.”

CONTENT DISCOVERY

Many major broadcasters and TV studios have vast libraries ripe for direct-to-consumer online sales. The challenge lies in determining which of these programs will appeal to modern consumers and for what reasons, without using employees to watch all of them in real-time.

Prime Focus Technologies’ CLEAR Vision Cloud has a cloud-based AI engine that can do this work across a number of search variables, and in “record time,” according to the company.

“There could be one AI engine that looks at identifying faces in the video,” said Muralidhar Sridhar, vice president of AI and Machine Learning for PFT. “Another one may look at signature sounds of, ‘let’s say, a person splashing through water,’ while a third searches for distinct objects. Best yet, what would take humans hours to achieve looking at a piece of content can be done by our AI in real time.”

Primestream’s Xchange platform uses AI/ML to power its content discovery tools, providing a wide range of search options in the process, according to Alan Dabul, director of product development for Primestream.

“You can narrow the search down not just to President Trump, but to those specific clips where he is talking about taxes,” he said. “You can then narrow the search further to those times when he is speaking about taxes in an office setting, and then see who is with the president in the shot at that time.”

SPORTS AND LIVE EVENTS

Sports and other live events are among the most labor-intensive productions for broadcasters, given how much content has to be created on the fly. Tedial’s SMARTLIVE metadata engine uses AI/ML to automate media management tasks associated with these productions; including metadata tagging, automatic clip creation and distribution during live events to digital platforms and social media. SMARTLIVE can also manage multivenue feeds and support multiple, instantaneous content searches to integrate archival footage into live broadcasts.

“SMARTLIVE allows the production team to create more content leading to increased fan  engagement and additional revenue, using the same budget and with the same team,” said Jerome Wauthoz, vice president of products for Tedial. “SMARTLIVE also connects directly to existing production environments so our customers can use their current infrastructure to ingest, edit and deliver content; no additional investment is necessary.”

CAPTIONING AND TRANSLATIONS

Another labor-intensive area where AI/ML is gaining traction is multilingual captioning. Using speech-to-test AI systems, vendors can automatically generate text captions from the content’s audio, and provision them in a range of languages within the same data stream.

“The algorithms are trained to learn from data in real-time, absorbing local terms and dialects for the optimal captioning experience,” said Brandon Sullivan, senior offering manager for IBM Watson Media. “As AI and machine learning training capabilities improve, local dialects, places and specific names, as well as the voices of individual speakers, will all be accurately captured. Down the road, this will not only transform closed captioning but also automated translation, video indexing, and more.”

Captioning and lip sync are two of the AI/ML technologies featured as part of Interra Systems’ BATON, a video QC platform. “With AI/ML, you can improve the accuracy and speed of captioning, which is a resource-intensive, time-consuming process,” said Anupama Anantharaman, vice president of product management for the Silicon Valley-based provider of video QC and monitoring technology. “It is also particularly effective at detecting ‘lip sync’; the alignment between the movement of lips onscreen and what is being said.”

Telestream’s Telestream Cloud includes captioning as its many cloud-based AI/ML-enabled offerings; the others being video transcoding for multiple delivery platforms and quality/compliance checks, according to Remi Fourreau, cloud product manager for the company.

“We use the speech-to-text capabilities of many cloud-based providers to generate accurate captions and subtitles in many languages,” Fourreau said. “This is an area where AI/ML really shines in doing the task accurately and efficiently.”

ENCO’s enCaption4 platform provides automated closed captioning for live and pre-recorded TV content in real-time, and combines AI-driven machine learning with a neural-network speech-to-text engine. In addition to newsroom rundown imports that teach unique words via AI, enCaption4 can be taught special words such as host and cast names, and local and regional terms. Other AI-driven enhancements improve the captioning of punctuation and capitalization.

“enCaption can accurately spell unusual words learned from ingested lists and scripts, and without creating speech pattern profiles for every speaker, said Ken Frommert, president of ENCO. “This is an important benefit for news operations automating and captioning speech from various anchors, reporters, meteorologists, and studio guests.”

COMPRESSION

Video compression has always been a balance between data rate reduction and video quality. Through AI- and ML-based cloud solutions such as its VOS360 Live Streaming Platform, Harmonic aims to strike this balance more effectively. 

“Our PURE Compression Engine uses AI/ML to improve the algorithms that manage video compression,” said Jean-Louis Diascorn, senior product marketing manager, who leads Harmonic’s AI/ML for video compression advances. “These improvements are achieved far quicker using AI/ML compared with using human engineers. We continue to make progress on the work that we presented at last year’s NAB BEITC and are now aiming to address the density aspect.” 

RECOMMENDATION ENGINES

Streaming services such as Amazon, Netflix and YouTube use AI/ML-enabled recommendation engines to mine their viewers’ current content choices, and use what they find to recommend similar programs that might be of interest. Vionlabs’ AI/ML-enabled Content Discovery Platform is designed to help broadcasters assess their own content libraries, to focus and enhance their Direct-to-Consumer sales online.

“High-quality data can help broadcasters understand so much more about their content and make better informed decisions throughout the content cycle,” said Marcus Bergström, CEO of the Swedish-based provider of video discovery technology. “One example of this is in content recommendations and providing broadcasters with a deeper understanding of how successful shows appeal to viewers. It could also help them automatically comply with regulations for post-watershed content.”

Last month, the company launched “Emotional Fingerprint API” to help media companies make better decisions based on AI-generated video data and insights. Emotional Fingerprint API uses computer vision and machine learning to generate sentiment-data, creating a unique personal viewer experience based on Vionlabs’ recommendation, according to the company.

Emotional Fingerprint API has been developed to measure thousands of factors during the screening of a video, including colors, pace, audio and object recognition, in order to produce an AI-derived fingerprint, frame by frame, that represents the emotional structure of content.

THERE ARE LIMITS

AI/ML-enabled systems are now fulfilling many roles in the TV production/playout stream. But they can’t do everything; at least not yet.

“For machine learning tools to work effectively, you need to continuously fine tune models and need large amounts of well-prepared data,” said Anantharaman. “There will be challenging situations where human intervention will be needed. Yet, for the majority of content, AI/ML can provide an extremely high level of accuracy.”

At the root of the recent attention given to artificial intelligence is what is known, at a global level, as the “digital transformation.” Although predominantly utilized in the context of business, digital transformation (DX) has broad reaching impacts to many areas not the least of which are the television media and entertainment industries. DX is reaching the public and business sectors, numerous organizational activities, business process management (BPM), social media, and institutions ranging from government through education.

This industry-wide digital transformation is fueled, in part, by the increased focus and capabilities of artificial intelligence (AI) and by the applications of machine learning (ML). DX, AI and ML are augmented services either in the cloud or occasionally on premises. AI touches applications available from resources including the Apple iPhone, Google AI, IBM Watson, and a growing set of others.

INTELLIGENT SERVICES

Multiple new services applicable to broadcast, news and sports are using AI as part of their content creation, recognition, assembly, and distribution engines. In the video industry, according to companies such as TVU Networks and Veritone, we are now experiencing a sea change in the way video content is searched, produced, distributed, and consumed. Cognitive computing is transforming the way we use and generate video. Video customization, a frequent output of AI, is enabling individuals to see continued augmentation in how video is consumed and where or how it is being distributed.

Functionally, workflows utilizing AI begin at the point files are ingested using smart content management features that extract metadata using prediction engines. The AI-based engines determine flow, subject matter, relevance, plus other attributes which then generate relevant search components with high accuracy. AI is at the top of the compute-centric food chain (Fig. 1).

Other applicable AI-based services include advertising verification and sponsorship efficiency to track and verify brand name mentions, logos and characterization. For news purposes, AI techniques will categorize stories, interviews, breaking news and features – at both the local and the national level. For sports, player recognition and accumulated play or scoring data is used to self-generate melds of the game or statistics with better relevance than humans can – and do it in real time.

BEYOND SIMPLE RECOGNITION

AI is not just about facial recognition or venue classification or text/speech interpretation. AI utilizes machine learning, but it is not data mining. For media applications, AI is a key supporting agent in search-engines which engage sophisticated machine language-based algorithms to, for example, catalog images and sound for applications of metadata extraction or collection. Reducing the amount of manual human interaction needed to sort or tag images and sound is both supplementing and adding new value to archives and catalog platforms – and AI now allows those applications to go much further.

Indexing – previously a manual post ingest task - can now begin the instant that the video ingest and production processes start. Based on derived metadata, indexing allows real-time search to be built immediately using AI. And that information can be instantly shared (permissions pending) with others including users and other AI-based databases and libraries.

BASED IN THE CLOUD

Many of these new indexing platforms are built entirely on a cloud-based model. Using voice and object recognition, both live and pre-produced video clips can be indexed right down to the exact frame. Where once the sophistication of automated indexing amounted to scene change detection alone, today intelligent resource supplements can use information collected from other analysis to ascertain, e.g., people in the frame, voice or action recognition of non-visible speaking humans, if the objects in the scene are animals, building, automobiles, etc., and where the scene was shot based upon databases and interpretations from other images. All this at a reliability in the 80-85 percentile on a first pass; and even better accuracy on future passes.

AI allows services to build libraries of information that “learn” from previous identifications which in turn improves accuracy and speeds up the indexing and cataloging time with each task. The more the systems see, collect and validate the content, the better and faster the solutions get. Fig. 2 identifies the more common applications of AI for US companies in 2016 and those applications are expanding rapidly.

THE APPEAL FOR BROADCASTERS

AI for broadcasting exploits the efficiency of employing machines that can interpret and understand audience demands by using data management and filtering techniques poised to analyze content for specific themes and then create original content applicable to the individuals, locations and interaction of those people and placed in the images. These applications are particularly useful for taking raw/live content and boiling it down to rough cuts that can be stitched together for rapid release to OTT or mobile devices.

AI concepts literally “open the floodgates for how programs are produced and distributed,” according to Paul Shen, CEO of TVU Networks. Removing heretofore “human-delegated” barriers from the production process, digital and broadcast programming departments can create a single centralized search engine for raw materials such as live or recorded feeds, across all channels.

Furthermore, the integration of AI helps media companies better target specific audiences with more appropriate programs and advertisements – not unlike what we’re experiencing with social media such as at Facebook and Twitter. These entities are all utilizing varying degrees of AI and ML.

CONNECTED STRATEGY

Digital transformation aides in creating and optimizing new capabilities by leveraging the possibilities and opportunities of new and emerging technologies. However, the DX journey needs a staged approach defined with a clear roadmap. Stakeholders need to envision a connected world that is beyond silos, with a strategy that tears down internal vs. external constraints and appraises end goals that will continue to move as DX becomes the “de facto end-point” position going forward.

The IoT is also helping drive this connected strategy concept whereby systems previously confined to developing high level designs, architectures, and plans are now shifting to media and content. The capability to fine tune operational activities for business or manufacturing are now being applied to everyday consumer products such as smart homes, intrusion security and autonomous vehicles. With the explosion of content being generated and an overall demand to see more, faster and better; AI must be applied to broadcast, media and entertainment in order to satisfy that thirst.

BEWARE OF HYPE

Unfamiliar terms bring new “marketing” opportunities filled with anxiety that can yield to confusion. DX, one of the latest buzz words, is no different. As with any emerging and/or disruptive technology, there are tendencies to look or select those tech companies’ who offer “sexy” products or claim to answer ‘all your needs’ in a single offering. Just be weary, because, generally speaking, digital transformation should be considered “industry-agnostic,” and it is likely to encompass many offerings in multiple scenarios.

DX should start with business goals, identification of challenges, an exploration of current and future customers or needs, and then apply those findings to the context of the organization.

Digital transformation usually happens at different speeds. DX creates new partnerships which mutually leverage their collective synergies to produce a sum value which is greater than their individual parts. DX can merge disruptive entities (technologies and organizations) into harmonious entities. But beware, simply selecting a single offering without understanding and anticipating the overall impact to the organization (and its partners) can be detrimental to the success of the DX challenge. Some providers have indeed been “disruptive” in the sense of forcing bigger players to adapt or decease – that is only part of the agenda.

Potential adopters of AI-based solutions can learn from the new start-ups as well as those technology success stories we then hear about. We are only beginning to see the depth and interaction which AI and ML can bring to workflows and operations like media asset management equipped with automated content recognition or content assembly.

Karl Paulsen is CTO at Diversified and a SMPTE Fellow. He is a frequent contributor to TV Technology, focusing on emerging technologies and workflows for the industry. Contact Karl at kpaulsen@diversifiedus.com.

SAN MATEO, CA.—Broadcast professionals responding to a survey at last month’s NAB Show foresee increasing reliance on AI and machine learning as well as hybrid cloud storage and a falloff on their reliance on tape-based storage in their future.

The survey, based on in-person interviews of more than 300 people at the show, revealed that 78 percent of broadcast professionals plan to use a combination of on-premise and cloud-based storage, also known as hybrid storage, to speed up media management.

Eighty percent plan to use AI and ML technologies to enrich metadata, the survey found. Among users of tape storage, 51 percent indicated they plan to move away from tape media over time.

[Read: The Next Big Step For AI? Understanding Video]

Among post-production professionals interviewed, 73 percent revealed they are frustrated with how desired media is located and retrieved and identified the issue as their primary storage challenge. Fifty percent said media management is more time-consuming today than it was three years ago.

Eight in 10 are thinking about using AI and/or ML technology to assist in tagging media, which indicates healthy interest in storage offering embedded rich metadata tags and metadata-bases search tools, the survey said

The survey also revealed a significant upturn in those who expect to use hybrid storage when compared to three years ago –78 percent versus 16 percent in 2015. Exclusive use of cloud storage appears to be headed lower, however, as 9 percent said they would be cloud-only in three years while 17 percent are today, it found. When it came to disk- versus tape-based storage as a primary storage medium, the former was clearly the preference, with 53 percent favoring disk and 32 percent choosing tape.

Among tape users, 51 percent said they plan to move away from tape in the next three years, the survey said.

Cloudian sells a scalable storage platform that consolidates, manages and protects enterprise data.