Virtual reality and its close relatives, such as extended reality, have been completely accepted into the workflow of high-end productions. Stepping into an LED volume, you can be transported anywhere from Antarctica to a planet embroiled in an imagined science fiction war. All it takes is a lot of pixels and a good imagination.

[Editor’s note: Here are a few of the examples of virtual reality technologies and case studies that were on display during the 2024 NAB Show. This article first appeared in the NAB Daily.]

MO-SYS VFX SOLUTION FOR DR. WHO ANNIVERSARY SPECIAL:

Mo-Sys Engineering recently announced that NearTime, its automated re-rendering service for on-set virtual production, was used to deliver VFX scenes for the “Dr. Who” 60th Anniversary Specials. In 2022, award-winning boutique studio Painting Practice, with CGI, animation and VFX specialists RealTime, approached Mo-Sys to develop a cost-efficient workflow for an episode of “Dr. Who” that required more than 250 VFX shots and was expected to be heavily VFX-dependent. Traditional VFX approaches would have been cost-prohibitive for this episode.

RealTime and Mo-Sys had previously collaborated on a Netflix production called “Dance Monsters,” which involved combining six cameras and eight monster characters in real time using the Unreal Engine and Mo-Sys VP Pro to create a live light-entertainment show. As part of this project, Mo-Sys and RealTime developed a pipeline for transferring precision camera and lens tracking data from Mo-Sys StarTracker through to post production to help automate and dramatically speed up the VFX workload.

By putting pre-visualization, on-set camera tracking, real-time pre-viz, NearTime rendering and automated VFX pipelines together, this VFX-heavy episode of “Dr. Who” was completed with stunning visuals at a fraction of the cost of traditional VFX.

XR STUDIOS “DANCING WITH THE STARS”: 

XR Studios, a Los Angeles-based facility with two LED volume studios, put together the season premiere opening for the Disney+ reality series “Dancing With the Stars” using a science fiction theme. Studio A has a 71-foot wide LED volume that is 20-foot tall and curved to provide a depth of 35 feet. Studio A also has an LED floor and a large movable LED panel that can be used to provide special lighting/background effects.

The overhead lighting grid uses many Kino Flo MIMIK light panels that can provide an LED ceiling. The MIMIK lights can also be put on stands to create lighting effects as needed in the production. For “Dancing With the Stars,” the LED volume was used to create an entirely fictional world, complete with spacecraft and hulking planets. The season premiere opening featured the two hosts of the show traveling in space through disco-themed planets. 

“They really wanted to get people looking as if they’re traveling through hyperspace, so we put two Kino Flo lights on lowboy stands right beneath them tilted up at their faces,” said J. T. Rooney, virtual production supervisor at XR Studios.

“The lights had hyperspace content that the talent could see and react to.” A little bit of realism in an unrealistic environment.

BEHIND THE SCENES OF THE INTEL EXTREME MASTERS — COLOGNE 2023:

In August 2023, the Intel Extreme Masters (IEM) was held in Cologne, Germany. Extreme sports fans filled LANXESS Arena for the final three days of the competition, dubbed “The Cathedral of Counter-Strike.” This event drew a sizable live audience and more than 700,000 fans online to watch the world’s top Counter-Strike: Global Offensive (CS:GO) teams go head-to-head.

IEM entrusted Rocket Surgery Virtual Productions to enhance the video with augmented reality (AR) graphics, immersing the audience and fans at home deeper into the action. Rocket Surgery used a range Ross Video’s augmented-reality technology, including the Voyager Render Platform based on the Unreal Engine, Ultrix hyperconverged routers, a Carbonite Ultra production switcher and an OpenGear frame equipped with a range of signal processing cards. Complementing this array of production room technology was the use of the Spidercam cable-based camera system to provide unique visuals and feed the tracking data required to power the AR images.

To address the challenges presented by the rapid pace of online gameplay, Ross developed a tailored solution combining DashBoard and Datalinq to integrate all their video game APIs and devices directly into the production workflow. Rocket Surgery then programmed a custom DashBoard control panel using the RossTalk protocol, that allowed operators to automatically merge inputs and data with AR graphics in real time.

RESOLUTION STUDIOS IS VIRTUAL THE CENTER OF CHICAGO: 

Resolution Studios is an 86,000-square-foot production complex used for a variety of projects including TV commercials, TV episodes and feature films, located in Chicago’s Near West Side. Planar recently added 400 Planar Venue Pro VX Series indoor LED video wall displays with a 1.9mm pixel pitch that, when fully assembled, form a curved, 80-foot-wide by 14-foot-high LED volume.

Designed for the demands of temporary LED video wall setups, the Planar Venue Pro VX features magnetically attachable cabinets and a quick-lock system for securing cabinets together. The full 80-foot by 14-foot volume can be reasonably reconfigured into an LED video wall that is 60-foot wide and 20-foot tall, or even broken up into multiple formations.

“We definitely have common configurations that we use over and over, but certain shoots require specific layouts,” said Todd Freese, senior vice president and chief technology officer with Resolution Productions Group. “For instance, the way we will set up an LED video wall for a car process shot is completely different than what we would do for a narrative. I don’t ever want to have the conversation of trying to convince a director to shoehorn their creative into a permanent installation that may not be right.”

WHISTLING WOODS EXPLORES INDIA’S CULTURE WITH VIRTUAL REALITY:

Whistling Woods International, a film and creative arts institute nestled in the heart of Bollywood/Mumbai Filmcity in India, is working on projects that combine the country’s rich cultural heritage with immersive storytelling. The Kumbh Mela, occurring every 12 years, is the world’s largest gathering of humanity, where more than 220 million people make their way to a specially constructed city on the banks of the Ganges River. The last Kumbh happened in 2019 and the Whistling Woods crew was there to shoot it.

“Kumbh Mela 2019” will surround viewers in a kaleidoscope of colors, culture and spiritual fervor, as millions of devotees converge for this extraordinary pilgrimage. Through the use of 360 video and virtual reality, this project promises to offer a unique perspective on the Kumbh Mela, highlighting its significance as a symbol of spiritual enlightenment and collective harmony.

With more than 100 hours of content banked, the outlines for what will go on to become the most comprehensive and immersive showcase of the Kumbh are ready and so is a 7-minute trailer. The entire comprehensive docuseries should be completed later this year.

Copyright NAB 2024. 

The beginning of the pandemic brought empty sports stadiums around the world. Even when the teams began to return to the playing field, the fans weren’t allowed back in the stands — at least not actual fans. Some production companies were able to create virtual fans, though similar technology was already in use before the COVID-19 breakout to enhance what the viewer was seeing at home. With the adoption of 5G wireless technology, smart phone users didn’t have to miss out either.

“We have been leveraging AI — artificial intelligence — to enhance the way we can bring the augmented reality experience to the audience,” said Thibault Baras, general manager of Dreamwall, which has been working with virtual and augmented reality technology for more than 20 years. “This has included different content, from virtual fireworks to virtual zeppelins that can fly overhead in a stadium.”

ENTERTAIN AND INFORM

This technology can entertain, inform and even be used to provide additional advertising opportunities for sports franchises. And with 5G, those in the stands will also be able to experience it as well. 

“Virtual studios are designed to amaze the viewer — even with growing expectations, amazing the viewer is still achievable,” said Gerhard Lang, CTO at Vizrt. “Photorealism is approaching a level where it is almost impossible to distinguish between a real studio and a virtual studio. This means our customers can take their viewers to locations and spaces with a quality previously only achievable by large movie productions in post. The technology exists, and it does not matter whether a game engine or native rendering is used for the production.”

The use of graphical technology has advanced so that augmented reality can blend near seamlessly with the physical, and this has been extremely important as the pandemic forced many to work remotely. If the sportscasters and broadcasters can’t go to an event, the technology can help better immerse them into the environment, which could immerse the viewer in the experience.

“Virtual technology and augmented reality help content creators to improve the viewing experience for the audience, especially when delivering large amounts of data or when creating information graphics. Because, if complex data are displayed in a visually attractive way, the audience can understand them better; and helps retaining audiences and explaining the concepts,” said Miguel Churruca, marketing director for Brainstorm.

“The COVID-19 pandemic has consolidated the virtual technology revolution. Virtual and remote solutions are here to stay and taking advantage of these will only help in satisfying an ever-growing demand for content,” said Churruca. “Viewers are demanding ever-more sophisticated graphics, and the increased demand for content creation, coupled with the contention of production costs and current mobility and distancing limitations, make augmented reality and virtual reality the perfect choice for any broadcaster or production company aiming to create high-end content.”

Game engines are also providing production teams with real-time feedback, and even allowing those teams to make informed creative decisions on set, explained James Uren, technical director at Mo-Sys Engineering.

“The technologies incorporated into our VP Pro and VP Pro XR solutions enable final pixel filming in LED volumes, previsualization for green screen filming and augmented computer graphics in live television broadcast,” Uren said. “These technical tools are giving directors, actors and production teams a tremendous boost and will be helping blur the lines even further between the real and the virtual components in a shoot, creating stunning visuals to engage and delight their audiences.”

CREATING MAGIC

We are at a point where graphics technology can create video magic like never previously seen. AMD is making inroads in those efforts with its third-generation AMD Ryzen Threadripper, which has been used by studios to create the wondrous and terrifying alike. 

“All good content is about the suspension of disbelief — and that is absolutely true whether you are watching a five-minute cartoon or a two-hour feature film,” said James Knight, visual effects and media and entertainment director for AMD. 

“It is about making the characters move accurately so that they seem like they are real,” he said. AMD’s technology helps in the production by making this work less time-consuming, by increasing the power the production teams are afforded.

Technology still is only half of it, said Vizrt’s Lang, who said that the final output still largely depends on the talent and skill of the artists working in the studio.

“This talent is sometimes underappreciated and underfunded in our industry,” said Lang. “In addition to beautiful backdrops, our solutions allow for stories to be told in understandable ways. Content is presented using the most efficient workflows in the industry, or by putting the anchor in the steering seat by enabling interactivity in the studio for them.” © NAB 2021

LOS ANGELES—Virtual Reality (VR) comes once again to the boxing ring Saturday (Nov. 14) with Fox Sports’ production of the Premier Boxing Champions (PBC) fight pitting Amilcar Vidal against Edward Ortiz.

Facebook Watch will present the VR fight via its Venues app to users of the Oculus Quest VR headset. Quest wearers tuning in for the fight will experience the match as if they were onsite witnessing the sights and sounds for themselves.  

The fight, the first of three VR events Fox Sports will produce for Facebook, will be shot with 4K cameras dedicated to the VR production. Oculus Quest users will be able to interact with one another in the VR environment and watch replays in virtual windows in their environment. 

“You as a virtual reality viewer become an avatar in this virtual environment,” says Michael Davies, senior vice president, Technical and Field Operations, at Fox Sports. “You are watching the fight from a kind of virtual amphitheater that you can move around in.”

“Just in front of you is this great big screen, essentially, that is the virtual reality programming. So, it is a mixture of VR and interactive elements,” he says. 

Boxing is particularly well-suited to good VR production because “the playing field is so small,” says Davies. 

“Years ago [in 2016], we did a boxing event and right at the beginning of our package we used NextVR,” he says. “We learned some things about shooting boxing in VR like the best camera placement.”

For Saturday’s fight, four 4K cameras will be used—one in a neutral corner, another ringside next to one of the judges and a third as a “wild cam” that can be moved around for shots of the boxers walking to the ring and other interesting viewpoints. The fourth, as of this writing, has not been assigned a location, says Davies.

The four cameras will feed a special VR production workflow that is separate from the linear show that will air on FS1 (Fox Sports 1) at 8 p.m. ET. Switching the VR production is similar to a standard video production with one important distinction: speed. “We do it a little bit slower as not to cause too much vertigo [for the VR viewer],” says Davies.  

“What works better with VR is to allow the scene play out in front of you versus trying to follow the action too much. It just takes your senses a little bit of time to acclimate to each shot,” he says. “So, we’d like to find a good shot and stay on it for a while and only switch when absolutely necessary.”

Graphics are added to the VR production as are replays from the linear production. The VR show director views a typical multiviewer with video that’s “like a very stretched picture” to call the shots for the show. Another production crew member is in the VR experience to ensure not only the quality of the production but also that it looks good in the Venues amphitheater, he says.

Audio for VR productions is particularly important. “We learned, and I think everybody learned when we were doing VR for the first time a few years ago, that there still very much is a need for storytelling in VR,” says Davies.

The sound mix, including audio from the VR camera mics, natural sound from the linear show and announcer audio, will help keep VR viewers grounded in the environment and move the story of the fight along. However, unlike the linear show, the production will have no commercials.

To keep VR viewers engaged during commercial breaks, Fox Sports announcers will engage with Oculus Quest wearers to answer questions and talk over the fight.

Fox Sports then will hand off the VR production to Facebook to place into the Venues environment, says Davies.

Producing a separate VR show is a rather low-profile endeavor and poses no threat of interfering with the linear show, notes Davies, who adds that the evolution of virtual reality has been impressive. 

“We’ve probably gotten about 15 or 20 VR shows under our belt over the years,” he says. “So, it’s satisfying to see that VR is marching along and improving with every manifestation. Boxing is a good place to showcase it once again and see what everybody thinks.”

(Editor’s note: Fox Sports and Facebook will also present two more PBC matches, to be determined, on Nov. 21 and Dec. 16.) 

Fremont, CA — August 29, 2018 - Blackmagic Design announced today that Micro Studio Camera 4K and Video Assist were used for 4K VR live streaming of a Jazztronik concert at Billboard Live Tokyo. The concert was streamed on YouTube and viewers could feel as though they were inside the concert venue by wearing a head mounted display.

The 4K VR live stream was organized by Kadinche, a developer of VR and AR solutions technologies, software development and web design.

Jazztronik, the music group who performed for the concert, is part of the Japanese company Styrism. Takayuki Morinaka, CEO of Styrism, said, “It was the first time for us to hold a live streaming concert. Of course it is best to have fans come to our concert and let them enjoy live performances, but we decided to give it a shot at making use of the cutting edge VR live streaming technology for our artists, having noticed that the technologies of live streaming and VR have been rapidly growing over the years.”

Two Micro Studio Camera 4Ks, with Entaniya 250° lenses, were placed at the concert venue, one for live streaming and the other for archive recording. Video Assist was used for monitoring and recording purposes.

“The Jazztronik concert was scheduled to last for one and half hours. We sought reliable cameras that could shoot for long hours and concluded that Micro Studio Camera 4K was the best option. The camera’s small size was especially helpful for this event as we had limited space to put a camera in. In contrast with the size of the camera body, the sensor inside the camera is surprisingly huge, giving you great picture quality in 4K. The 250° lenses and Micro Studio Camera 4K were the best combination because both of them support micro four thirds mount. Micro Studio Camera 4K satisfied every condition this project required.”

“We reduced the frame rate on our software to 4K 20fp to achieve stable live streaming. As affordable as it is, Micro Studio Camera 4K is a wonderful camera with great 4K image quality. Combined with Video Assist, you can even see the video captured by a fisheye lens, which is not possible with common VR devices,” said Dai Akiyama, chief strategy officer of Kadinche.

“I think that VR streaming is best for concerts and events. You can let as many people as you want experience the performance without inviting them to actual venues. Concert halls have limited seats, but with VR streaming anyone can enjoy and see artists at the front row. We are hoping to continue using Blackmagic products for projects of VR and stereoscopic video streaming,” he concluded.

Press Photography

Product photos of Micro Studio Camera 4K, Video Assist and all other Blackmagic Design products are available at www.blackmagicdesign.com/media/images.

About Blackmagic Design

Blackmagic Design creates the world’s highest quality video editing products, digital film cameras, color correctors, video converters, video monitoring, routers, live production switchers, disk recorders, waveform monitors and real time film scanners for the feature film, post production and television broadcast industries. Blackmagic Design’s DeckLink capture cards launched a revolution in quality and affordability in post production, while the company’s Emmy™ award winning DaVinci color correction products have dominated the television and film industry since 1984. Blackmagic Design continues ground breaking innovations including 6G-SDI and 12G-SDI products and stereoscopic 3D and Ultra HD workflows. Founded by world leading post production editors and engineers, Blackmagic Design has offices in the USA, UK, Japan, Singapore and Australia. For more information, please go to www.blackmagicdesign.com.

IROQUOIS, Ontario — Ross Video this week announced a forthcoming user interface release that improves dynamic, intuitive control over virtual productions.

The new version, UX 4.0, will update the Ross Video UX Xperience, a control center application that users can customize with an intuitive touchscreen graphical user interface. It integrates seamlessly with tracking systems, key products and real-time 3-D rendering engines, including Ross Video’s Xpression and Frontier, the company said.

[Read: Ross Updates Acuity Switcher With 12G Single-Link Connectivity]

“UX 4.0 is building on the success of previous UX versions,” said Gideon Ferber, director of marketing product management for Ross Virtual Solutions. “We’ve added tools to make the production smarter and more dynamic while keeping operation as straightforward and user-friendly as ever.”

The new release enhances the functionality and connectivity of various components, including UX Track, UX Server, workflow capabilities and tracking and mounts. Ross also is adding a new Visual Logic feature and an Unreal4 plug-in with UX 4.0.

The Visual Logic feature is a new quadrant that lets users create sophisticated behaviors by dragging and dropping parameters on a canvas and connecting them in a visual, node-based way. While no previous coding experience is needed to use Visual Logic, the feature offers an extensive set of tools to create intricate behaviors, the company said.

The Unreal plug-in gives users a means of feeding data as strings, integers, floats, vectors, drive animations and meshes from UX into the blueprint layer of the Unreal engine, it added.

[Want more information like this? Subscribe to our newsletter and get it delivered right to your inbox.]