It’s become a virtuous circle: Advances in graphics and editing technology lead to higher quality and better-looking content, especially on mobile devices. The audience becomes accustomed to these improvements, which drives the need for further advances in improved quality and content appearance.

Advanced graphics and editing are the unsung heroes of modern video production and the trends and technology that support these heroes will be on full display on the 2025 NAB Show floor in Las Vegas.

As consumers become accustomed to high-quality visual content across all their devices, broadcasters are feeling the pressure to meet these expectations with their own content to stay competitive, according to Sarah Jones, senior technical solutions specialist for Blackmagic Design, Americas.

“This demand means broadcasters are upgrading production equipment and investing in technologies capable of delivering higher resolutions and cinematic images,” Jones says. “While this could impact production budgets, today’s technologies are more accessible than ever before, packing professional features into affordable budgets, so we’re seeing a reduction in the barrier to entry when it comes to the use of cutting-edge technology.”

Standing Out
Uniqueness is also becoming more important in the video graphics and editing sector. With content providers serving out so many channels in an ever more-crowded video universe, being different and eye-catching is vital for building audiences.

As a result, “a significant trend we’re seeing at ENCO Systems is the need for both customizable channel branding and dynamic graphics insertion,” says Bill Bennett, media solutions and
accounts manager for the company.
“Broadcasters and streamers often have the same content feeding several different channels, each requiring its own branding, so they rely on our ClipFire video playout system to separately brand each signal.

“Furthermore, these same folks like to host cost-efficient automated channels with dynamic, compelling graphics which change with real-time data [think weather or sports], so we’ve been able to help them do that, too,” Bennett adds.

Of course, serving out all these extra channels costs money, which is a commodity that content providers cherish. That is why economy is a third trend influencing the graphics and editing sector, with vendors doing their best to devise solutions that deliver advances without emptying the bank.

“​​Whether an individual content creator or a large broadcaster, many are looking to achieve more with less,” Vizrt Chief Marketing Officer Chris Black tells TV Tech. “They’re looking for workflows and technology that can support them creating more content, faster, with multiple aspect ratios for dual outputs, leveraging their current investment without the need to add in new tech, or hire new people. Two big focuses we are seeing are the use of dual-aspect ratios for social media content alongside linear or multiregion
advertising to monetize content across various demographics.”

At the same time, production teams are seeking new ways to scale up their operations that are inherently flexible and efficient. “As a result, we are seeing the further democratization of media technology,” Black says. For instance, “cloud-native and web-based HTML5 graphics platforms with intuitive UI are offering existing graphics templates for individuals to use, increasing efficiency in time to air.”

What They’ll Be Showing
Blackmagic Design will have two booths in South Hall Lower, (SL216 and SL8016), and will showcase cameras, DaVinci Resolve postproduction software and hardware, its wide range of SMPTE-2110 IP video products, the ATEM family of live production switchers, streaming solutions, network storage and more, according to Jones.

“Pairing our URSA Cine Immersive and DaVinci Resolve provides an end-to-end workflow to support these evolving interactive needs, providing the means to capture and edit 8K, 3D video with a 180-degree field of view,” Jones says. “The Blackmagic URSA Cine Immersive is the world’s first commercial camera system designed to capture Apple Immersive Video, and DaVinci Resolve provides the ability to edit Apple Immersive Video for the first time.”

When it comes to automating time-intensive tasks, DaVinci Resolve Studio’s Neural Engine-backed AI tools can significantly speed up workflows. “This comes into play, for example, with our facial recognition technology, which can automatically detect, track and organize footage based on an individual’s face, eliminating the need to search and organize manually,” Jones says. “Similarly, smart reframe tools intelligently resize and reposition footage to fit various aspect ratios, facilitating content repurposing across platforms.

ENCO Systems will showcase its ClipFire flagship multichannel playout and ingest platform which can service a lot of needs (and channels) from a single box, according to Bennett.

“We’ll be showing where we think the future of broadcast graphics are headed, thanks to our Qimera Virtual Production produc,” Bennett says. “Built on the massively popular Unreal Engine, Qimera generates stunning truly three-dimensional graphics which humans can interact with and walk around in 3D space, complete with graphics (and backgrounds) that move as the humans do in-camera, making for some incredibly compelling video. We’ll also be showing the new MOS newsroom automation hooks we’ve built into Qimera recently, to automate the data that’s making those 3D dynamic graphics so compelling.”

Also in the West Hall, Vizrt will demonstrate Viz Connect Audio, “which offers unparalleled connectivity, visibility, and control of audio sources on an NDI network,” Black says. “We are particularly excited about Viz Connect Audio, and how it seamlessly integrates with our flagship TriCaster, the TriCaster Vizion.”

This year’s show will be the first time that the multi-award-winning TriCaster Vizion will be demoed at NAB Show, offering the latest in IP connectivity, configurable SDI I/O, powerful switching, audio mixing and state-of-the-art graphics. “Show attendees can also expect to see advancements in Viz Engine’s seamless integration with Unreal Engine,” Black says. “Building on benchmarks for ease of use and performance, Vizrt will demonstrate—via Viz Arc—enhanced control of the latest versions of Viz Engine 5 and Unreal Engine 5.”

Vizrt also plans to display its latest advances in immersive graphics, cloud, automation, and AI. “We aren’t just showing technology,” Black concludes. “We are showing real-world solutions that solve everyday production challenges across three experience areas and several solution pods.”

SAN DIEGO—Sony Electronics has announced the latest version (Version 2.0) of its virtual production software, Virtual Production Tool Set, will be available winter 2024.

Sony's Virtual Production Tool Set is a pair of software applications, the Camera and Display Plugin and the Color Calibrator, that improve the virtual production workflow for broadcast and cinema – making it easier to get predictable, high-quality results. Both applications are free to download while some features will be available under a paid license.

The Camera and Display Plugin allows users to simulate a specific camera’s movement and settings on screen in a virtual space, prior to physical production with a LED wall. Version 2.0 of the Camera and Display Plugin expands the list of supported broadcast and cinema cameras to include BURANO, FR7, HDC-3500, HDC-5500, and HDC-F5500 adding to VENICE series. The Camera and Display Plugin allows productions to plan shots as well as leverage the tool’s improved custom Moire alert tool, Sony said. 

Version 2.0 also supports Epic Games' Unreal VCam, making it possible to intuitively control cameras in virtual space using an iPad. Additionally, Sony offers a paid license for Live Camera Sync, which can instantly read changes in the settings of VENICE cameras and lenses on set and reflect them in the Unreal Engine Sony virtual camera settings.

Color Calibrator is a Windows 10 application that creates correction LUTs iv to achieve high color reproduction in Virtual Production stages and performs color correction (color calibration) of the installed LED walls. The LUTs calibrates the LED wall to the Sony camera color space to ensure that the colors visualized before filming match the colors captured while filming. Color Calibrator Version 2.0 supports color calibration via the newly added camera models as well as a user guide function that makes it easier to perform relevant adjustments. The Color Calibration tool is free but creating and exporting calibration LUTs is available through a paid license v.

This latest version of the Virtual Tool Set will be on display at IBC 2024 at the Sony booth, in Hall 13.

For more information on the latest version of the Virtual Production Tool Set, please visit our website: https://sonycine.com/virtualproduction

Missed any of our recent product news? Here is TV Tech's weekly wrap-up of new products, services and deployments we published in our newsletter between August 5 to August 9. The stories are listed from the newest on Friday August 2, day by day back to our stories from Monday August 5. 

Telemundo Takes VR Studio Tech from Miami to the Paris Olympics

Telemundo's virtual reality studio in Miami uses Unreal Engine and Chyron to provide a "photo-realistic look" that shows small details of the Paris landscape like the "wind in the trees".

Haivision To Feature Live Video Contribution at IBC 2024 

The company will show the latest developments in its live contribution ecosystem.

Imaginario AI To Demo Video Indexing, Search, Curation At IBC 2024

The company will show its multimodal artificial intelligence solution in the AI Tech Zone.

AI: The Unsung Hero of Olympic Sports

Can AI sustain Olympic revenue during and between Games?

A New Take on Reflectors: Cine Reflect Lighting Systems

The need for better tools has inspired lighting innovation.

Disney+ Launches Nat Geo Immersive Environment for Apple Vision Pro

Disney+ has announced that subscribers to the streaming service can now experience a new immersive environment from National Geographic with the Disney+ app on Apple Vision Pro.

Wheatstone to Showcase Blade 4 at IBC 2024

The expanded capabilities of the 4th generation AoIP Blade 4 include high speed transport protocol to routable audio codecs, audio processing, scripting, and studio control in 1RU.

Telos Alliance To Feature Next Generation AERO DTV Audio Processors At IBC Show

The AERO.20, AERO.200 and AERO.2400 include new features like AoIP support via AES67.

Finding The ‘Boun-dairies’ For AI In Media

TV channels are using AI-generated presenters to read the news. The question is, will we trust them?

M&E’s Embrace of AI Dampens Potential for Damage Mitigation

Amid the hype, industry struggles to establish guidance and guardrails.

Google TV Streamer Launches at $99.99

The new Google TV Streamer offers an AI-powered, personalized 4K HDR streaming with Dolby Vision Dolby Atmos and a smart home experience.

2024 IBC Show to Focus on AI, Business and Tech Innovation

IBC President Michael Crimp previews the upcoming gathering.

Mediaproxy To Show AI-Powered Captioning, Transcription for LogServer At IBC 2024

LogServer now supports AI-Media Technologies LEXI Recorded and Skyline Communications DataMiner.

Kramer Buys ZeeVee To Expand AVoIP Portfolio

ZeeVee’s expertise in AVoIP will expand Kramer’s product lineup and accelerate growth.

EVS Invests in TinkerList

EVS becomes a shareholder in the broadcast software provider as part of its strategy of offering simplified live production solutions.

XR Extreme Reach, Roku Expand Streaming Ad Partnership

The expanded collaboration allows advertisers to link all ads on the Roku platform to performance outcomes.

Arkona, Manifold Partner To Show EASY-IP At IBC 2024

EASY-IP combines arkona’s BLADE//runner and manifold’s CLOUD routing, processing and multiview tool.

MIAMI & PARIS —As the Paris Olympics games head into their final weekend, much of the tech talk has been about the use of AI technologies in the production of the games. But the games have also seen notable advances in a number of other areas, including virtual reality technologies and remote production solutions that have played an important role Telemundo’s Spanish-language coverage. 

During this year’s coverage, Telemundo’s main studio has been a photorealistic virtual set in Miami that looks like it is in the Trocadero, offering a stunning view of the Eiffel Tower. This technology, which was used successfully in previous events like the 2020 Tokyo Olympics and FIFA World Cups and developed in-house, has seen further improvements for the Paris Games and has allowed Telemundo to cover the Olympics with talent based out of its Studio L in its Miami headquarters. 

"For the Paris 2024 Olympics, we wanted to have the best possible spot in Paris, right in front of the Eiffel Tower,” explained Fabian Albaraccin, director real-time graphics at Telemundo. “We know that securing such a location is nearly impossible, so we recreated `Trocadéro Square’ with a stunning view of the Eiffel Tower and the city. “To accomplish this, we began working four months ago to ensure we could showcase the City of Light in all its splendor. We paid close attention to every detail to present a beautiful set to our audience.”

The virtual set, which was developed in-house by the Set Fabrication and Real-Time Graphics teams, combines green screen studios, physical scenic pieces, advanced camera tracking, and computer-generated graphics. The result, Telemundo executives say, creates an immersive Parisian experience without the high costs and weather concerns of on-location production. Real-time adjustable lighting and weather, along with augmented reality graphics also enhance Telemundo’s storytelling.

For graphics, the setup uses four cameras, the Stype/RedSpy tracking systems, the Blackmagic/Ultimatte chroma keyer, and is powered by Epic Games/Unreal Engine 5.3 and Chyron/Prime VSAR for graphics. 

Manashi Mukherjee, vice president of technical production at Telemundo explained that the set is located in Studio L, which is on the south side of the building where the larger soundstages are located. 

“We normally shoot our series in the studios on that side of the building along with special productions such as our recent `Billboard: Women in Music awards show,'” Mukherjee said. “On a regular production week, Jose Diaz-Balart’s 11am MSNBC show on weekdays and on `Saturdays for Nightly News' on NBC also originate in the virtual studio. During football season, we have Telemundo’s `Sunday Night Football' production as well in Studio L. For the Paris Olympics, we have been using Studio L seven days a week live on-air for Telemundo’s coverage.” 

“Universo, our sister network, is using the regular Sports (“Deportes”) Studio A which is also airing simultaneously from TC,” Mukherjee added. 

Wally Rodriguez, vice president of post-production noted that “we’ve had a dedicated studio for virtual production for almost five years, which makes it easy for both the technical operations team and the sports team because they have been using the set for a while now. Our camera operators, lighting personnel, audio technicians, producers, directors, anchors, and others are all familiar with the equipment and virtual production processes for day-to-day on-air operations. The use of moving lights allows the team to reset and use several various setups without moving any physical instruments.”

While the solution has been relatively easy to use, it did have to be adapted for broadcast. 

“Since Unreal Engine is a gaming software, we need to adapt it for broadcast purposes,” explained Albaraccin. “We are working closely with Chyron to provide a tailor-made solution that ensures Unreal Engine is stable and fully integrated with our state-of-the-art facility. Additionally, we create custom panels to manipulate Unreal Engine while we are on air. Because this is not originally a broadcast solution, we must bridge the gaps and create our own workflows. Using these panels hides the technicalities of creating and triggering the graphics for the shows so that producers can focus on content. Collaboration is key in these types of projects, and while our team handled the technology, we worked with Deportes producers and the Telemundo Scenic design team to ensure that the creative vision was met.”

The result has been an extremely realistic presentation that has greatly enhanced their coverage, Mukherjee said. 

“The virtual set has allowed us to showcase a much larger space than we would have used in Paris,” Mukherjee added. “The technology makes it possible to feel like the talent are there. The team has fine-tuned miniature details to give it a photo-realistic look that hasn’t really been used for broadcast television yet. The use of the Unreal engine makes it so that small details such as wind in the trees, cars in the background and even people walking around in the Eiffel Tower, can be used to make it feel realistic. The crew and the talent almost forget they’re on a green screen set, because they are looking at the composited images. Camera operators are moving the jib in the same way they would if they were in Paris. Advantages are lower costs for labor and materials than a regular set, sustainability, lots of flexibility in framing for example, we can move the set, instead of moving a camera and the ability to create a scene without being limited by the cost of materials or physical studio space.”

The set has also proved to be adaptable for a variety of events, in part, because it is relatively easy to train staff. “It was rather straight-forward to train the teams as most positions continued working the way they are used to as we have been using this space for about five years now,” Mukherjee said. “Camera operators, directors, producers, and talent see the shots with fully rendered backgrounds, so they don’t have to imagine what a shot will look like when they’re just looking at green screens.”

That will also help Telemundo in the runup to the 2024 elections and for other events. 

"We have a special that will be filmed in the studio after the Olympics wraps, and then outside of the normal news and sports production, we will be incorporating this studio into the upcoming election coverage,” Mukherjee said. “We haven’t seen the final graphics plan yet, but there will be a virtual component in play.”

OSLO—Pixotope has announced the availability of Pixotope 24.1.0 and is reporting that the release includes the full integration of Unreal Engine 5.4, which opens up the ability to design motion and production graphics directly in Pixotope. 

With this integration Pixotope now offers a full broadcast graphics solution for simplified and streamlined augmented reality (AR), virtual studio, extended reality (XR) and character generator (CG) workflows with Unreal Motion Design, the company said. 

In addition, this release features an arsenal of innovative updates designed to empower broadcasters with the tools to enhance creativity and reduce complexity in virtual production workflows. The result is that it is easier and more efficient than ever for broadcasters of all sizes to push the boundaries of immersive storytelling to grow and retain their audience to drive new revenue opportunities, the company reported. 

“Broadcasters are constantly seeking ways to differentiate their programming. With Pixotope 24.1.0, we're providing them with powerful tools to leverage Unreal graphics in their live broadcasts without complicating their workflows,” says Gideon Ferber, senior vice president of product at Pixotope. “For broadcasters, these creative capabilities translate into unique revenue opportunities. For example, they can integrate clever advertising seamlessly within programming, effectively monetizing 3-dimensional space that previously had little to no value.” 

With the integration of Unreal Engine 5.4, Pixotope 24.1.0 delivers industry-leading motion design capabilities, allowing for complex 2D motion graphics directly within the engine. This enables broadcasters to take their on-air branding and visual storytelling to new heights, creating more engaging and immersive viewer experiences. With this integration, broadcasters can now easily produce highly detailed and photo realistic visual effects, enhancing both storytelling and visual appeal.

In addition to the enhanced creative capabilities of Unreal Engine 5.4, Pixotope’s broadcast graphics solution is easy for broadcasters to onboard. With Pixotope and Erizos Studio under one roof, users get simplified, streamlined, and familiar template-based workflows. This allows broadcasters to control multiple graphics engines and various aspects of their show simultaneously, integrating with newsroom control systems using the MOS protocol and supporting automation-driven playouts. These template-based workflows enable broadcasters to easily deliver compelling visual stories to viewers, seamlessly integrating real-time AR, XR, and virtual studio graphics into their live broadcasts.

“Broadcasters today face the dual challenge of producing high-quality content while managing tight budgets and complex workflows,” Gideon adds. “By automating complex tasks and reducing the need for extensive technical resources with ease of use updates, broadcasters can optimize their workflows to get engaging content out faster and more often." 

Pixotope 24.1.0 includes the following new features and capabilities that dramatically improve efficiency and reduce complexity:

• Hybrid XR Wall Capabilities that enable 2D Media Playback on LED walls, expanding the use cases to include both 2D and 3D graphics, offering a flexible solution to changing production needs.
• Chroma Screen Actor defines areas where the keyer will be active, ensuring smooth transitions between virtual and physical sets and improving the quality of hybrid studio productions by managing keying more effectively.
• Other creative enhancements available in Pixotope 24.1.0 via Unreal 5.4 support include: Character Rigging Enhancements; Heterogeneous Volume Rendering; Motion Matching for Animators; Rendering Improvements 
• Automatic Delay Detection automatically calibrates delays between video and tracking, saving time and reducing errors by simplifying synchronization processes.
• Data Mapper allows users to bind external data to graphics without needing complex blueprints, reducing technical complexity and easing data integration into visual elements.
• Embedded Tracking data directly within the engine, eliminates the need for additional equipment (especially useful for remote productions), saving time and reducing the need for expert calibration.
• Simplified Color Pipeline simplifies the setup process for managing color spaces and profiles, ensuring consistent color quality and easy management of color settings.
• Template-Based Workflows simplify the creation process, enabling users to produce high-quality visuals more easily.
• XR/Video Wall Improvements offer new options for mapping media inputs and generating manual twins for XR walls, enhancing flexibility and control over XR setups.
• Server Failover improves redundancy and safety for pipelines, ensuring productions continue smoothly even if a server fails.

Pixotope 24.1.0 is available immediately to all Pixotope customers. More information on Pixotope release 24.1.0 is available here. 

Epic Games has released a major new version of Unreal Engine, which has become a mainstay in virtual production, that includes new features and improvements to performance, visual fidelity, and productivity that the company said will benefit broadcasters, game developers and other creators in the media and entertainment industries. 

Notable new features for the new Unreal Engine 5.4 include features for broadcast (Unreal Engine Motion Design Mode, previously known as Avalanche) that are already being used in sports production, and new capabilities for virtual production (including Unreal's virtual camera app, Unreal VCam, on Android for the first time).

Epic Games is positioning the Unreal Engine Motion Design Mode, which was used by CBS in producing the Super Bowl earlier this year, as a game-changing environment meticulously crafted to empower motion graphics designers with precisely tailored tools. 

Motion Design Mode offers a new layout is designed to ensure seamless workflow initiation and sustained productivity, the company said. Equipped with specialized tools for the 2D canvas and an enhanced world outliner featuring intuitive hierarchies, Motion Design Mode provides a comprehensive suite including 3D shapes, cloners, effectors, modifiers, animators, and features for non-linear animations. This dedicated mode is engineered to elevate the motion graphics design experience, offering a robust and user-friendly platform for designers to bring their creative visions to life. 

Over 300 companies and 550 individuals have been beta-testing Unreal Engine Motion Design Mode since last year. CBS Sports used numerous Unreal Engine tools to deliver the stunning Super Bowl LVIII motion graphics package designed around the Las Vegas strip. CBS Sports also used Unreal Engine to kick off the show opener with an XR promo shoot at the MSG Sphere and live AR graphics during the game. 

Another notable upgrade in 5.4 is Unreal VCam for Android, which is now available on Google Play. 

This is the same app that has been known as Live Link VCAM and only available on iOS until now. The new release of Unreal VCam will enable virtual production-ready cinematography in real time from any Android device.  Unreal VCam enables filmmakers to translate traditional concepts like camera lensing, framing, and movement into Unreal Engine’s digital realm. 

It allows users to record realistic camera takes that can be further fine-tuned in editor to create elegant final compositions when assembled in Unreal Engine’s Sequencer. Review early set environments through the lens, previs shot cameras before a shoot, capture real takes during principal photography, create a new shot in post, or all of the above.

A video overview of the many new features in UE 5.4 is available below. here: https://www.youtube.com/watch?v=ltQfA5Wqdik

The Walt Disney Company and Epic Games have announced a major alliance to collaborate on all-new games and what the companies are calling a new "entertainment universe." As part of the agreement, Disney will also invest $1.5 billion to acquire an equity stake in Epic Games. 

The deal is notable in that it marks Disney’s biggest ever investment in games and will provide Disney with a new platform for showcasing its characters, brands and stories. It also builds on a longstanding relationship between Disney and Epic Game's Fortnight that goes back to Epic Games’ participation in the 2017 Disney Accelerator program.

While Disney and others have backed away from some of their metaverse efforts, the agreement will see the creation of a metaverse-like “open entertainment universe” with Fortnight that will offer a multitude of opportunities for consumers to play, watch, shop and engage with content, characters and stories from Disney, Pixar, Marvel, Star Wars, Avatar and more.

In this entertainment universe, the companies said players, gamers and fans will be able to create their own stories and experiences, express their fandom in a distinctly Disney way, and share content with each other. This will all be powered by Unreal Engine.

In a call with analysts, Disney CEO Bob Iger said: “Our new relationship with Epic Games will create a transformational games and entertainment universe that integrates Disney's world-class storytelling into Epic's cultural phenomenon, Fortnite, enabling consumers to play, watch, create and shop for both digital and physical goods.”

“This marks Disney's biggest entry ever into the world of video games and offers significant opportunities for growth and expansion,” Iger said during Disney’s fiscal Q1 2024 earnings call. “The new immersive universe will allow fans to unleash their own creativity and experience the Disney stories in worlds that they love in groundbreaking new ways.”

Iger also stressed the growing importance of games among younger audiences and the opportunities the deal will create to reach them. “Younger audiences in particular are huge consumers of video games,” he said. “In fact, among millennials, Gen Z and Gen Alpha, a significant amount of time spent on screen-based platforms is playing video games. This new universe from Disney and Epic provides us with a tremendous opportunity to not only meet more consumers where they are, but to allow more audiences to cultivate a bond with Disney's iconic brands and franchises, including Marvel, Star Wars and much more.”

The agreement expands an existing relationship between Disney and Epic Games. In the past, Disney and Epic Games have engaged hundreds of millions of players through Fortnite content integrations, season collaborations, in-game activations, and live events, including the Marvel Nexus War with Galactus, which drew more than 15.3 million concurrent players.

“Disney was one of the first companies to believe in the potential of bringing their worlds together with ours in Fortnite, and they use Unreal Engine across their portfolio,” said Tim Sweeney, CEO and founder, Epic Games. “Now we’re collaborating on something entirely new to build a persistent, open and interoperable ecosystem that will bring together the Disney and Fortnite communities.”

In the past, Unreal Engine has been used to produce assets and content across the Disney portfolio including in the development of video games like Kingdom Hearts 3 and Star Wars Jedi: Survivor; in cinematic editing and animation for film and streaming; and in the creation of more than 15 Disney Parks attractions like Millennium Falcon: Smugglers Run at Star Wars: Galaxy’s Edge.

With more than 3 billion video game players worldwide, Disney also said the agreement will further expand its existing games business, which shifted to a licensing business model in 2016. 

Currently, Disney is a leading games licensor working with developers and publishers, including on the best-selling superhero game of all-time, Marvel’s Spider-Man. Licensed games from Disney garnered more than 150 award nominations, wins and other accolades in 2023, including multiple Game of the Year nominations for Marvel's Spider-Man 2. Disney mobile games have 1.5 billion global installs, and to date, nine Disney games franchises have each grossed more than $1 billion in sales. In the U.S., the world’s largest games market, licensed titles from Disney regularly hit the annual top-10 best-sellers list, the company said. 

MIAMI—Grup Mediapro’s Mediapro US subsidiary has launched what it is billing as the largest virtual production center in South Florida.

The new facility includes a giant LED wall - 24 meters wide, 7 meters high and with over 1,000 panels - at its Miami studios. 

The opening is part of a multi-year agreement signed between Mediapro US and BrandStar, one of the longest-standing production companies in the United States, and represents a notable expansion of Mediapro's operations into the virtual production sector.

As part of the deal, BrandStar designed, constructed and installed the LED wall and will offer support services for the operation of the virtual production system.

This audiovisual installation will use Unreal Engine software, a 3D creation tool allowing the live blending of real environments with digitally created elements. 

Grup Mediapro reported that the launch is part of an ambitious plan to equip its main production centers with virtual production technology over the course of 2023-24. In this first phase of its rollout, the group plans to open centers in Miami, Barcelona, New York and Madrid.

In Barcelona, the virtual production set will be launched in late 2023. This studio will be located in 22@, Europe’s largest urban audiovisual production hub, and will combine 235 square meters of LED panels with green screens. 

In the third quarter of this year, the group will add a new, 1,500 square meter virtual production center in New York. The second phase of this strategic plan will be completed with the 2024 launch in Madrid of another center. In later phases, Grup Mediapro will deploy this technology in all its production centers.

In a statement announcing the virtual production studio, Miami-Dade County mayor Daniella Levine Cava said that “the film industry is one of Miami-Dade’s most special economic drivers. Expanding the footprint of Miami-Dade County’s film and production capabilities is one of our key strategic initiatives, and we are honored to see this new partnership bring a first-of-its-kind studio to our county. I look forward to seeing a new level of production coming into Miami-Dade thanks to this new partnership.”

BrandStar founder and CEO Mark Alfieri added that “the company is delighted to be opening a facility in Miami-Dade County. The ability to combine our experience in producing over 300 TV episodes and 1,000 commercial video projects a year with that of the Grup Mediapro team will create new opportunities for the TV and film industry.”

Each year, Mediapro US produces more than 5,500 hours of TV content and operates 10 TV channels. With a 200-strong team of staff, Mediapro US provides production services to numerous TV shows for clients such as CNN, ESPN, beIN Sports, HolaTV, Hemisphere Media Group and CONCACAF.

NEW YORK—Deloitte and Sinclair Broadcast Group have announced that they plan to launch a new metaverse sports fan community experience that will leverage Epic Games' Unreal Engine to explore the future of gameplay and the fan experience through the metaverse.

The community will launch the week of March 6th.

The fandom community was built by Deloitte's Unlimited Reality practice, which brings together 3D computing, artificial intelligence, Web3.0, immersive experiences and advanced connectivity to help clients create business value. The community makes use of Epic's Unreal Engine — a powerful 3D creation tool that brings robust experiences to life in real time.

Sinclair is billing the partnership as an important next step in Sinclair's plans to deepen engagement with its viewing community, drive new revenue streams through experiential gaming and help redefine sports viewership and experiences through the metaverse. 

The community is designed to build on the new ways fans are consuming sports-related content, as well as the desire from fans for more immersive content and greater engagement with teams, Sinclair said.

It is also designed to engage fans beyond simply viewing of live games, providing a metaverse experience to engage fans during post-season, pre-game, post-game and eventually, during the game. 

"By bringing together Sinclair's vision of virtual metaverse engagement with Deloitte's deep client and sports industry knowledge, metaverse know-how and break-through solutions and ecosystem partnership, we plan to deliver an enhanced fan experience that caters to the unique communities we serve," said JR McCabe, chief business officer, Consumer Products, Sinclair Broadcast Group.

"In Deloitte's recently published '2023 Sports Industry Outlook', it is clear the blending of physical and virtual worlds is accelerating in the industry, providing fans with more ways to access and consume sports content than ever before," said Dan Helfrich, chair and chief executive officer of Deloitte Consulting LLP. "We are excited about the opportunity to team with Sinclair Broadcast Group to augment the viewing experience by harnessing the power of virtual worlds for broader community accessibility and deeper fan engagement."

Learn more about Sinclair's virtual community here.

FREMONT, Calif.—Disney employed Blackmagic Design’s Ultimatte 12 real-time compositing processors and HyperDeck Studio 4K Pro broadcast decks as part of its massive virtual production workflow for the live-action version of “Pinocchio,” the company said this week.

Starring Tom Hanks as Geppeto, the CGI retelling of “Pinocchio” required one of the largest virtual production workflows ever used for a feature film, which included several massive sets at Cardington Stage in London.

Jim Geduldick, director of virtual production and senior vice president of Dimension North America, supervised virtual production. He and his team integrated Simulcam, the live compositing of virtual objects with live action, with Unreal Engine’s real-time 3D CGI creations, allowing any crew member or actor to control or react to virtual objects in real time. 

While this approach has been used on previous projects, none were of the scale necessary for “Pinocchio,” Blackmagic Design said.

To address the massive undertaking, Geduldick created several virtual production carts built around Ultimatte 12 advanced keyers, Ultimatte Smart Remote 4 and HyperDeck Studio 4K Pros that could be moved anywhere they were needed on set, the company said. 

“No one had ever tried to do this across multiple sets for three months straight,” said Geduldick. “There was so much happening each day, and we had to make sure we could get a cart wherever and whenever they were needed. We could not have done that without Ultimatte and HyperDeck.”

Initially, Geduldick used two Ultimatte 12s per Simulcam feed, but with software improvements Blackmagic Design made to the keyer, he was able to use a single Ultimatte 12 per Simulcam setup by the end of the production, the company said.

“The workflow used Simulcam with layers of augmented CG layers and Ultimatte 12 as the keyer. The images were then integrated into Unreal Engine in real time. We could do front plates and back plates through a single Ultimatte, and we relied on the Ultimatte’s hardware-based processing to send a real-time key to Unreal,” he said.

“You can’t beat Ultimatte’s hardware-based real-time keying. Offloading to hardware saves so much data use that is needed elsewhere on set.”

The workflow relied on the HyperDeck Studio 4K Pros to create custom playlists along with basic playback and recording. This enabled a workflow in which virtual elements and specific lighting setups could be played back through Unreal Engine, the company said.

“The Simulcam and Ultimatte workflow was amazing,” he said. “The DP, VFX supervisor and camera operators would know where they were landing because they could see it using Ultimatte. Lighting and atmospheric decisions could be done right on set

“Virtual production only works if you can react and make creative decisions in real time, and that is what the Blackmagic Design products allowed.”

More information is available on the company’s website.

Grass Valley shared its vision of the future of media production and announced both a multiyear strategic collaboration with Amazon Web Services (AWS) and the leveraging of Epic Games’ Unreal Engine for real-time rendering with the GV AMPP (Agile Media Processing Platform). The company also made the move to no longer charge for Playout on AMPP.

CEO Andrew Cross said Grass Valley had set out on a journey three years ago to become the company creating the future of media production. He called AMPP a cloud-based software-as-a-service (SaaS) platform for producing live workflows launched in 2020 the infrastructure of the future, compatible with every standard.

Cross said the company was quite happy with AMPP adoption, and noted that Brazilian media giant Grupo Globo had already produced and streamed more than 50 live sports and entertainment events, with more than 200 planned for the year ahead

Although details were light, the enhanced collaboration with AWS will help Grass Valley leverage AWS’s secure, reliable and globally available infrastructure to allow customers to create, produce and monitor their media supply chains where they want and at scale.

Grass Valley also said it was working with Epic Games to bring Unreal Engine, a real-time 3D creation tool for photorealistic visuals and immersive experiences, to GV AMPP. Unreal Engine on AMPP will allow customers to use the rendering system natively with all Grass Valley video technology.

The company also said it will no longer charge for playout on GV AMPP, making the cloud-native system accessible to many more media and entertainment companies. The core AMPP Playout functionality will be available free to customers from June 1.

 ©NAB 

LAS VEGAS—On the heels of the public release of Unreal Engine 5, executives from the Unreal Engine team at Epic Games announced at the 2022 NAB Show some of the ways that Unreal Engine is being used by broadcasters and some of the partnerships they are forging with a wide variety of technology providers. 

“Sports, news, award shows, and weather broadcasts have been boosting audience engagement with use of AR, and we’re now increasingly seeing Unreal Engine being used beyond that for real-time motion graphics as well,” explained Bernt (BK) Johannessen, UE business director, broadcast and live events, Epic Games. “The Unreal Engine team first came to NAB in 2017, and at the time had about a dozen partnerships in the broadcast space. Since then we’ve seen a massive uptick and now have more than fifty broadcast ecosystem partners who have invested heavily in real-time workflow integrations powered by Unreal Engine.”

Unreal Engine announced that some of the companies servicing the broadcast industry with planned or existing Unreal Engine integrations include: Accuweather, Adobe, AJA Video Systems, AMD, AOTO, ARRI, ARwall, Aximmetry, Autodesk, Black Magic Design, Bluefish, Boris FX, Brainstorm, Brompton Technology, Chyron, Disguise, Dreamwall, Grass Valley, HP, HTC, Leyard, Matrox, Maxon, Megapixel, Mo-sys, Ncam, NEP, NVIDIA, OptiTrack, Panasonic, Pixel Power, Pixotope, Planar, Red Giant, ROE Visual, Ross Video, RT Software, SMT, Sony, Stype, Trackmen, TV-Graphics, Ventuz, Vicon, Vizrt, Weather Channel, WeatherMetrics, WTvision, Zeiss Lenses, Zero Density, Zlense and more.

Top broadcasters have also recently deployed Unreal Engine to achieve remarkable achievements in both live events and produced programming, the company said. The deployments include: BBC Sports built an entirely virtual environment for the winter Olympics in Beijing; The Weather Channel is using augmented and mixed reality to deliver groundbreaking weather reports; Karate Combat created dynamic virtual environments for martial arts competition shot entirely on a stage; Fox’s Alter Ego brought live virtual avatars to life for a high concept singing competition; the Atlanta Braves are boosting fan engagement in the metaverse; the Carolina Panthers executed an augmented reality in-stadium spectacle; and NBC’s The Voice featured a live volumetric captured performance of the Coldplay x BTS hit song “My Universe”.

The company also reported that the recently released Unreal Engine Sample for broadcast and live events illustrates how to design, develop and play out numerous animation elements in a live broadcast scenario leveraging the power and flexibility of Unreal Engine.

The sample, called “Hype Chamber,” is a tested and proven project developed for the Rocket League Championship Series esports broadcast with prebuilt rigging that can be adapted to any team sports production. The sample provides broadcast producers and motion graphics artists with a template for building a real-time broadcast package with the ability to quickly swap out logos, high-quality assets, modify color palettes, and repurpose assets for immersive spaces, live events or any additional needs outside of broadcast.

More information on how Unreal Engine is transforming broadcast, live events, film and television is available here. 

The Hype Chamber broadcast sample project can be downloaded here. 

CARY, N.C.—Epic Games announced a $2 billion round of funding, with existing investor Sony Group Corporation as well as KIRKBI, the family-owned holding and investment company behind The LEGO Group, each investing $1 billion. 

The new funding will help Epic Games expand its work on building the metaverse and position the company for further growth by creating new social entertainment exploring the connection between digital and physical worlds, the companies said.

Epic continues to have only a single class of common stock outstanding and remains controlled by its CEO and founder, Tim Sweeney. 

“As we reimagine the future of entertainment and play we need partners who share our vision. We have found this in our partnership with Sony and KIRKBI,” said Tim Sweeney, CEO and founder, Epic Games in announcing the funding. “This investment will accelerate our work to build the metaverse and create spaces where players can have fun with friends, brands can build creative and immersive experiences and creators can build a community and thrive.” 

“As a creative entertainment company, we are thrilled to invest in Epic to deepen our relationship in the metaverse field, a space where creators and users share their time.” said Kenichiro Yoshida, chairman, president and CEO, Sony Group Corporation.

Yoshida also highlighted Epic Games' work in virtual production: “We are also confident that Epic’s expertise, including their powerful game engine, combined with Sony’s technologies, will accelerate our various efforts such as the development of new digital fan experiences in sports and our virtual production initiatives.”

Last week Epic Games’ Unreal Engine announced that its Unreal Engine 5 was now available to download. “With this release, we aim to empower both large and small teams to really push the boundaries of what’s possible, visually and interactively,” the company said.  “UE5 will enable you to realize next-generation real-time 3D content and experiences with greater freedom, fidelity, and flexibility than ever before.”

Some in the production industry had called Unreal Engine 5 a game changer prior to its release. 

In an interview last fall with TV Tech, Greig Fraser, who won a Primetime Emmy Award for Outstanding Cinematography for his work on the Disney+ original and recently won an Oscar for his work on “Dune.” said: “When we did ‘The Mandalorian,’ we used off-the-shelf technologies like the Unreal game engine that were created for games, not TV production. Now you have technologies that are purpose-built for virtual production with many more capabilities. We’re still in the early days but the [upcoming] release of Epic Games’ Unreal Engine 5 will be a game changer.”

Epic's post-money equity valuation is $31.5 billion. The closing of the investment is subject to customary closing conditions, including regulatory approvals.

IZMIR, Turkey—Zero Density has unveiled RealityHub 1.2, a massive update to its production software that lets broadcasters manage everything from real-time graphics to robotic cameras all in one place with a web browser. 

With the upgrade, users can integrate most on-set equipment, automation systems and external data sources to RealityHub straight out of the box, or simply leverage Zero Density’s open-source SDK to develop their own custom integrations, the company said. 

“In a single application, RealityHub grants our users the ability to configure, compile, launch and playout projects, while providing time-saving ways to optimize their workflow,” reported Andrew Maloof, senior graphics system engineer at The Weather Channel. “Centralizing these tools into an intuitive, easily-accessible web interface changes the game for our artists, engineers and graphics operators.”

Whether it’s for interactive touchscreen applications or large video wall productions, RealityHub’s HTML5 interface can now also be used to access Unreal Engine thanks to RealityHub 1.2’s new Unreal Engine Vanilla support, Zero Density announced. 

Combined with Reality Engine’s capabilities for virtual studio and broadcast graphics, the new RealityHub release will make it fast and easy for broadcasters to utilize Unreal Engine directly from the same platform they use to manage everything else in their studio.

“RealityHub was designed to help customers like The Weather Channel, Warner Media and Eurosport by bringing traditional standalone control systems into a web browser,” explained Kuban Altan, co-founder and vice president of R&D at Zero Density. “This lets teams manage entire broadcast setups without having to worry about long installation processes. With RealityHub 1.2’s Unreal Engine Vanilla support, we are now opening up to a future where RealityHub technology is more accessible than ever before, giving Unreal Engine creatives in broadcast, education, live events and any other industry the power to control every element of their production from a single hub.”

Other key benefits, according to Zero Density, include:  

• Stress-Free Teamwork — Promote collaboration with a web-based control interface that everyone can access on their tablet, laptop or desktop.
• Real-Time Data Integration — Keep your audience in the loop by natively connecting to data sources that will automate the display of weather, election, sports or financial information — as well as keep updating in real time.
• Native Integration with Broadcast Ecosystems — Integrate with Newsroom Control Systems such as Octopus with MOS or any proprietary protocol. REST API also allows users to integrate with automations and build third-party integrations.
• Free Community Edition — Supported by Zero Density’s recent MegaGrant from Epic Games, the Community version of RealityHub is feature-complete, but only accessible to one user at a time.

The Community Edition of RealityHub 1.2 will be available to download for free before the end of 2021. For updates on the release, visit www.zerodensity.tv/products/reality-hub.

WHITE PLAINS, N.Y.—SMPTE has announced that its Rapid Industry Solutions (RIS) On-Set Virtual Production Initiative (OSVP) has received an Epic MegaGrant from Epic Games. 

The virtual production initiative is designed to bring together technology and thought leaders to quickly assess challenges relating to virtual production, establish goals, and make interoperability specifications, workflows, and best practices open and free to the industry along with supporting educational courses and webcasts. 

More than 40 companies and individual contributors from across the industry are collaborating as part of the RIS OSVP community to achieve this goal. 

The $150,000 Epic MegaGrant will help SMPTE speed up that initiative, which is its first Rapid Industry Solutions (RIS) effort.  

"This Epic MegaGrant is a wonderful validation of the work we've done so far, the model we're using to deliver results, and our ultimate goal of making those results open and free to the industry," said SMPTE Executive Director Barbara Lange. "We formed the RIS On-Set Virtual Production Initiative to address an important and specific industry need, and this support will add to our momentum and accelerate our work. Speaking on behalf of the SMPTE RIS OSVP community, I can say we're all tremendously grateful." 

SMPTE developed the RIS model to address rapidly evolving technologies and their application across the media and entertainment industry. 

With the goal of providing a clear, readily accessible framework for on-set virtual production, as well as much-needed resources for education and professional development, the SMPTE RIS OSVP Initiative is designed to help creatives, technologists, and executives to explore the full potential of powerful, real-time, 3D creative tools, including Epic's Unreal Engine. 

David Morin, industry manager, media & entertainment at Epic Games noted that "establishing interoperability standards and best practices for film and episodic content creators is essential to making virtual production workflows accessible to all storytellers. This initiative from SMPTE marks an important step forward for virtual production, a technique centered around use of Unreal Engine that has revolutionized the creation of interactive storytelling and entertainment.  We are pleased to support this effort through Epic MegaGrants, accelerating SMPTE's mission to drive the industry forward."  

More information about the SMPTE RIS OSVP Initiative is available at smpte.org/rapid-industry-solutions.

More about Epic MegaGrants can be found at unrealengine.com/en-US/megagrants.

Additional TV Tech coverage of virtual productions and the SMPTE initiative can be found here and here. 

Virtual production techniques have a long history in Hollywood but have really exploded in popularity in the last two years, thanks to travel restrictions during the pandemic and a host of tech improvements. 

Boom time: Much of the current interest stems from the use of large LED walls and gaming engines to create the fantastic virtual worlds seen in “The Mandalorian.” 

“When we did ‘The Mandalorian,’ we used off-the-shelf technologies like the Unreal game engine that were created for games, not TV production,” said Greig Fraser, who won a Primetime Emmy Award for Outstanding Cinematography for his work on the Disney+ original. “Now you have technologies that are purpose-built for virtual production with many more capabilities. We’re still in the early days but the [upcoming] release of Epic Games’ Unreal Engine 5 will be a game changer.”

Many others agree: “There are hundreds of stages going up around the world and we’re directly involved in dozens of them already,” said Adam Schmidt, executive vice president at Planar, a supplier of LED panels that recently launched a virtual production initiative Planar Studios.

Gaming engines drive new production capabilities: “What is really exciting is [how] the film industry [is starting] to truly understand the power of game engine technology in the context of world-building,” explained Miles Perkins, industry manager of film and television at Epic Games. It recently released a new version of its gaming engine, Unreal Engine 4.27, with the highly anticipated full release of 5.0 set for early 2022. 

The 4.27 includes a host of new features designed to improve “quality, efficiency and ease of use” for virtual productions, Perkins said.

“When Epic Games releases the full version of Unreal 5, the current limits imposed on the cinematic architecture will be lifted, allowing for more realism,” said A. J. Wedding, co-founder of Orbital Studios.

Final pixel shooting: “The ability to shoot basically anywhere without travelling to that location was the primary driver” of interest in virtual productions early on but as the technology has improved, “the potential to reduce VFX time and costs is a big one,” explained Sean Sheridan, regional technical manager for the Americas at Brompton Technology, which has emerged as a major player in the virtual production tech space. 

That is improving the image quality of video shot of actors in front of LED walls without a lot of post-production work. “Final pixel-shooting [is] really hot,” added Michael Geissler, CEO at Mo-Sys Engineering, which recently released the VP Pro 4.27 software that has been used on virtual productions for Amazon and Netflix. 

Expertise is as important as technology: NEP, which recently launched its NEP Virtual Studios with the acquisition of Prysm Collective, Lux Machina and Halon Entertainment, has made providing production expertise a central part of their strategy, explained Zach Alexander, co-president of Lux Machina NEP Business unit at NEP Virtual Studios. 

To help the industry cope with the complexities of a virtual production, SMPTE has embarked on a high-profile virtual production initiative. “It has become very, very clear that education is a big obstacle, not only among professionals wanting to use the technology but in terms of training people to handle virtual productions,” said Kari Grubin, a consultant who is spearheading the project, which is focusing on such areas as interoperability, best practices and education. 

Educators are also getting involved. Sheridan College’s Screen Industries Research and Training (SIRT) Centre in Toronto, for example, has built a new Virtual Production Innovation Hub. “We really want to be a sandbox where creatives can advance the technologies and better understand how they can use it,” said SIRT production lead Jason Hunter.

New cameras shoot new virtual worlds: More innovation is coming from traditional vendors of production tech. The SIRT Center, for example, relies on a number of technologies from Blackmagic Design, including the URSA Mini Pro 12K. 

ARRI has also jumped into the virtual production sector in a big way. It created some smaller experimental stages in Munich and Burbank to integrate their products into those from other companies and launched a new virtual production studio in U.K. in the summer of 2021. The ARRI solutions group also worked with the German production company to set up a virtual production stage in Berlin.

© NAB 2021

LONDON—Vizrt has announced details of the advanced virtual set graphics ecosystem that was used on BBC Sport’s “Pres 2” set and a home studio for BBC Sport in Salford, UK.  

John Murphy, creative director and head of graphics for sport at the BBC explained that “we had a small studio space at Media City which was not being used as much as we would have liked so we decided to convert it into a green screen space. With the virtual design and rendering technology we now have a studio that has 5 different presenting positions and is able to house a variety of our sports output.” 

The conversation used Vizrt’s new Viz Engine 4 with its integrated Unreal Engine 4 render pipeline and Fusion Keyer, coupled with the Mo-Sys StarTracker 6-axis optical tracking system and set designs from BK Design and Lightwell. 

In April 2021, BBC Sport kicked off its use of the all new Vizrt ecosystem, which will broadcast a variety of sporting tournaments and productions throughout 2021, including Match of The Day, Euros 2020 and the Summer Games.

Gerhard Lang, CTO for Vizrt, noted that “Vizrt was proud to participate in BBC Sports’ celebration of the UEFA Euro 2020 championships and the amazing summer games, and for us the BBC set stands out both in looks and functionality. With the hybrid Unreal 4-Viz Engine 4 studio we were able to solve many challenges for the BBC. The studio provided an operator friendly workflow that enables content change and provides viewers with meaningful information throughout the show.”

More information on the latest iteration of Viz Engine and its integration with Unreal Engine is available here. 

Think virtual production is the preserve of James Cameron? The confluence of games engines with faster PCs, LED backlots and off-the-shelf tools for anything from performance capture to virtual camera is bringing affordable real-time mixed reality production to market.

Cameron saw this coming, which is why he has upped the ante to where no filmmaker has gone before and decided to shoot the first  “Avatar” sequel as a virtual production under water. Not CG fluids either, but with his actors holding their breath in giant swimming pools.

“The technology has advanced leaps and bounds at every conceivable level since 'Avatar' in 2009,” says Geoff Burdick, SVP of Production Services & Technology for Cameron’s production company Lightstorm Entertainment.

Massive amounts of data is being pushed around live on the set of “Avatar 2,” Burdick says. “We needed high frame rate (48fps) and high res (4K) and everything had to be in 3D. This may not be not the science experiment it was when shooting the first 'Avatar' but … our setup is arguably ground-breaking in terms of being able to do what we are doing at this high spec and in stereo.”

This is just the live action part. Performance captured of the actors finished two years ago and is being animated at Weta then integrated with principal photography at Manhattan Beach Studios.

“Avatar 2” may be state of the art, but it’s far from alone. Most major films and TV series created today already use some form of virtual production. It might be previsualization, it might be techvis or postvis. Epic Games, the makers of Unreal Engine, believe the potential for VP to enhance filmmaking extends far beyond even these current uses.

Examined one way, virtual production is just another evolution of storytelling—on a continuum with the shift to color or from film to digital. Looked at another way it is more fundamental, since virtual production techniques ultimately collapse the traditional sequential method of making motion pictures.

The production line from development to post can be costly in part because of the timescales and in part because of the inability to truly iterate at the point of creativity. A virtual production model breaks down these silos and brings color correction, animation and editorial closer to camera. When travel to far flung locations may prove challenging, due to COVID-19 or carbon neutral policies, virtual production can bring photorealistic locations to the set.

Directors can direct their actors on the mocap stage because they can see them in their virtual forms composited live into the CG shot. They can even judge the final scene with lighting and set objects in detail.

What the director is seeing, either through the tablet or inside a VR headset, can be closer to final render—which is light years from where directors used to be before real-time technology became part of the shoot.

In essence, virtual production is where the physical and the digital meet. The term encompasses a broad spectrum of computer-aided production and visualization tools and techniques, which are growing all the time, meaning that you don’t need the $250 million budget of “Avatar 2” to compose, capture, manipulate and as good as publish pixel perfect scenes live mixing physical and augmented reality.

GAME ENGINES

The software at the core of modern, graphics-rich video games is able to render imagery on the fly to account for the unpredictable movements of a video-game player. Adapted for film production, the tech consigns the days of epic waits for epic render farms to history.

The most well-known is Epic’s Unreal Engine, which just hit version 5 with enhancements intended to achieve photorealism “on par with movie CG and real life”.  A virtualized micropolygon geometry, for example, frees artists to create as much geometric detail as the eye can see. It means that film-quality source art comprising hundreds of millions or billions of polygons can be imported directly into the engine—anything from ZBrush sculpts to photogrammetry scans to CAD data—and it just works. Nanite geometry is streamed and scaled in real-time so there are no more polygon count budgets, polygon memory budgets or draw count budgets; there is no need to bake details to normal maps or manually author LODs; and there is no loss in quality.

Epic also aims to put the technology within practical reach of development teams of all sizes by partnering with developers to offer productive tools and content libraries.

It’s not the only game in town. Notch has a new real-time chroma keyer, which when combined with its automated Clean Plate Generation produce “fantastic” results with almost no setup or tweaking while providing all the features you’d expect such as hair, liquid handling and hold-up mattes all within less than a millisecond.

ILM, which uses a variety of engines, also uses proprietary real-time engine Helios, based on technology developed at Pixar.

“The Jungle Book,” “Ready Player One” and “Blade Runner 2049” all made use of Unity Technologies’ Unity engine at various stages of production thanks to custom tools developed by Digital Monarch Media.

For example, on “Blade Runner 2049,” director Denis Villeneuve was able to re-envision shots for some of digital scenes well after much of the editing was complete, creating a desired mood and tempo for the film, using DMM’s virtual tools.

Games engines rely on the grunt power of GPU processing from the likes of Intel, Nvidia and AMD, which has got exponentially faster to enable real-time compositing.

DIGITAL BACKLOTS

The use of video walls in film and TV goes back at least a decade as a light source projecting onto Sandra Bullock and George Clooney in “Gravity.”

More advanced versions playing pre-rendered sequences were deployed by ILM on “Rogue One: A Star Wars Story” and its follow-up “Solo” and during a sequence set on a Gotham metro train in “Joker.”  A system is also being used on the latest James Bond, “No Time To Die.”

The most sophisticated set-ups combine LED walls (and ceilings) with camera tracking systems and games engines to render content for playback not only in real-time, but in dynamic synchronicity with the camera’s viewpoint. The result allows filmmakers to stage scenes with greater realism than with a green or blue screen and with far more chance of making decisions on set.

“The big change has come with more powerful GPUs combined with games engines providing the software for real-time rendering and ray tracing,” says Sam Nicholson, who heads Stargate Studios. “When you put that together with LED walls or giant monitors, we think that at least 50% of what we do on set can be finished pixels.”

For HBO comedy-thriller “Run,” the production built two cars outfitted to resemble an Amtrak carriage on a soundstage in Toronto. These rested on airbags that could be shaken to simulate movement. Instead of LEDs, a series of 81-inch 4K TV monitors were mounted on a truss outside each train window displaying footage preshot by Stargate from cameras fixed to a train traveling across the U.S.

“It’s a smaller scale and less expensive version of Lucasfilm’s production of 'The Mandalorian,' but the principal is the same,” explains cinematographer Matthew Clark. “It effectively brings the location to production rather than move an entire production to often hard to access locations.”

Any light that played on the actor’s faces or on surfaces in the train had to be synchronized to the illumination outside the windows, otherwise the effect wouldn’t work.

“It was important to line up the picture so when you’re standing in the car your perspective of the lines of train track and power lines has to be realistic and continuous. If the angle of the TV screen is off by just a few degrees then suddenly the wires of a telegraph pole would be askew. When we needed to turn the car around to shoot from another angle the grips could flip all the monitors around to the exact angle.”

LED displays are measured in pixel pitches (the distance in millimeters from the center of a pixel to the center of the adjacent pixel) are narrow enough for the images to be photographed. The panels are capable of greater brightness, higher contrast ratios and displaying 10-bit video.

Rental companies in the U.S offering LED screens or monitors include PRG and  Stargate Studios, and in the U.K., disguise and On Set Facilities, both of which also have operations in LA.

OSF advises that the bigger the pixel the more light it outputs onto your subject, which means very fine pixel pitches may not be optimum for filming. The pixel pitch resolution of the LED screens used on “The Mandalorian” was 2.8.

OSF is set up as a fully managed virtual production studio covering in-camera VFX (LED), mixed reality (green screen) and fully virtual (in-engine) production.

It has a partnership with ARRI and also has its own virtual private network connected to the Azure cloud for virtual production. StormCloud enables remote multi-user collaboration in Unreal Engine powered by Nvidia Quadro technology. Entry points currently set up in London and San Francisco are being tested by “a number of Hollywood Studios and VFX facilities,” says the facility.

CAMERA TRACKING

Another essential component is the ability to have the virtual backlot tracked to the camera movement by a wireless sensor. This means that as the cinematographer or director frame a shot, the display, which is often the main lighting source, adjusts to the camera’s perspective. That’s no mean feat and requires minimal to zero latency in order to work.

Professional camera tracking systems from Mo-Sys and N-Cam are the go-to technologies here, but if purely filming inside a games engine there are budget ways of creating a virtual camera.

To create raw-looking handheld action in his short film “Battlesuit,” filmmaker Haz Dulull used DragonFly, a virtual camera plugin (available for Unity, UE and Autodesk Maya) built by Glassbox Technologies with input from Hollywood pre-viz giants The Third Floor.

Another option is the HTC VIVE tracker, which costs less than $150 and has been tested at OSF. “If you want to shoot fully virtual, shooting in engine cinematic is amazing with a VIVE as your camera input,” it sums up. “If you want to do any serious mixed reality virtual production work or real-time VFX previz, you are still going to need to open your pocket and find a professional budget to get the right equipment for the job.”

PLUG-IN ASSETS

The Rokoko mo-cap suit can stream directly into UE via a live link demoed by OSF. The facility explains that the suit connects over the wireless network to the UE render engine and into Rokoko Studio where OSF assigns the suit a personal profile for the performer. It then begins streaming the data into UE by selecting the Unreal Engine option in the Rokoko Studios Live Tab (a feature only available to Rokoko Pro Licence users). The system is being refined at OSF with tests for facial capture in the works.

Reallusion makes software for 3D character creation and animation, including iClone and Character Creator 3D. The Unreal Live Link Plug-in for iClone creates a system for characters, lights, cameras and animation for UE. The simplicity of iClone combined with UE rendering delivers a digital human solution to create, animate and visualize superior real-time characters. It’s also free for indie filmmakers; details are here.

Character Creator includes a plug-in called Headshot, which generates 3D real-time digital humans from one photo. Apart from intelligent texture blending and head mesh creation, the generated digital doubles are fully rigged for voice lipsync, facial expression and full body animation. Headshot contains two AI modes: Pro Mode & Auto Mode. Pro Mode includes Headshot 1,000-plus sculpting morphs, Image Mapping and Texture Reprojection tools. The Pro Mode is designed for production level hi-res texture processing and ultimate face shape refinement. Auto Mode makes lower-res virtual heads with additional 3D hair in a fully automatic process.

OSF ran this through its paces, using Headshot to automatically create a facial model, which was animated within iClone 7 using data from actors performing in Rokoko mocap suits streamed live to iClone allowing real-time previews and the ability to record animations. OSF also used Apple’s LiveFace app (available for download on any iPhone with a depth sensor camera) and its own motion capture helmets to capture the facial animations. The next part of the pipeline is to transfer the assets over to UE with the Unreal Engine LiveLink plugin and Auto Character set up plugin, which creates skin textures in the same way as Epic Games’ digital humans.

VIRTUAL PRODUCTION ON A BUDGET

British filmmaker Dulull made the animated sci-fi short “Battlesuit” using Unreal Engine, on a skeleton budget and team of just three, including himself.

Rather than creating everything from scratch, they licensed 3D kits and pre-existing models (from Kitbash3D, Turbosquid and Unreal). Dulull animated the assets and VFX in real-time within Unreal’s sequencer tool.

They retargeted off-the-peg mocap data (from Frame Ion Animation, Filmstorm, Mocap Online) onto the body of the film’s main characters. For facial capture they filmed their actor using the depth camera inside an iPad and fed the data live into UE.

“We had to do some tweaks on the facial capture data to bring some of the subtle nuance it was missing, but this is a much quicker way to create an animated face performance without spending a fortune on high end systems,” Dulull says.

Powering it all, including real-time ray tracing, Dulull used the Razer Blade 15 Studio Edition laptop PC with Nvidia Quadro RTX 5000 card.

Every single shot in the film is straight out of Unreal Engine. There’s no compositing or external post apart from a few text overlays and color correction done on Resolve.

“If someone had said I could pull off a project like this a few years ago that is of cinematic quality but all done in real-time and powered on a laptop, I’d think they were crazy and over ambitious,” he says. “But today I can make an animated film in a mobile production environment without the need for huge desktop machines and expensive rendering.”

This story originally appeared on TVT's sister publication Creative Planet Network.

April 8, 2019 - Las Vegas, NV, NAB Show - Epic Games is at NAB 2019 to showcase the latest Unreal Engine broadcast technology integrations and workflows. Unreal Engine broadcast solutions and use cases are being demonstrated at the Las Vegas Convention Center April 8-11 at the Studio Xperience Booth #SL3824 alongside more than 50 appearances with customers such as NFL Network, FOX Sports, The Weather Channel, and more.

Epic Games is at NAB on the heels of accepting the first Technology and Engineering Emmy for Unreal Engine in the 2017-2018 category, “3D Engine Software for the Production of Animation” during last night’s 70th Annual Science & Technology Emmy Awards hosted by the National Academy of Television Arts & Sciences.

Where to see Unreal Engine at NAB 2019:

— April 9, 2:00-2:30PM, N419, Epic Games Technical Product Manager Andy Blondin will present "Blurring the Lines Between Games, Film and Television: Epic Games Case Study"

— April 8-11, 11:00 and 11:30AM daily, #SL3824, Unreal Engine workflow presentations from Weather Channel's Michael Potts, Girraphic's Grant Werle, NFL Network's Pat Lee, Fox Sports' Zac Fields, MYREZE's Bjorn Mireze and Ben Grafchik of Sneaky Big

— April 8, 2:00PM, #SL3824 an interview with Kim Libreri, Epic Games CTO providing a general overview of Unreal Engine technology

— April 9, 1:00PM #SL3824 an interview with Andy Blondin, Epic Games Technical Product Manager, covering Unreal Engine broadcast features

— April 10, 3:30PM #SL3824 an interview with David Morin, Head of LA Lab, Epic Games talking about industry technology and Unreal Engine

— April 11, 12:00PM #SL3824 an interview with Ryan Mayeda, Technical Product Manager, Epic Games presenting about Unreal Engine and VFX pipelines

A range of Unreal Engine integration partners will be exhibiting at NAB, several of whom will showcase virtual production and real-time broadcast workflows. Those partners include: AJA Video Systems, AR Wall/AMD, Avid, Blackmagic Design, Bluefish 444, Brainstorm, ChyronHego, Maxon, Mosys, Ncam, NewTek, Procyc, Ross Video, Stype, The Future Group, Vizrt, wTVision, and Zero Density, along with many other customers featured in panels and presentations during the week.

Epic Games recently launched Unreal Engine 4.22, bringing powerful new features to content creators and storytellers. The release introduces ray tracing and a host of virtual production tools, including a new UI for the Composure compositing tool; support for OpenColorIO (OCIO) color profiles; Sequence Take Recorder for faster iteration and review of takes; multi-user collaborative editing features, and much more.

About Unreal Engine

Epic Games’ Unreal Engine technology brings high-quality games to PC, console, mobile, AR and VR platforms. Creators also use Unreal for photorealistic visualization, interactive product design, film, virtual production, mixed reality TV broadcast and animated entertainment. Follow @UnrealEngine and download Unreal for free at unrealengine.com.

About Epic Games

Founded in 1991, Epic Games is the creator of Fortnite, Unreal, Gears of War, Shadow Complex, and the Infinity Blade series of games. Epic's Unreal Engine technology, which brings high-fidelity, interactive experiences to PC, console, mobile, AR, VR and the Web, is freely available at unrealengine.com. The Epic Games store offers a handpicked library of games, available at epicgames.com. Follow @EpicGames for updates.