Immersive sound is the final journey to the complete auditory experience. Stereo began that journey with the ability to reproduce sounds that would align more with our hearing, but at the onset, audio practitioners seemed to define stereo as two channels of monophonic sound.

I will always remember my first records that separated the instruments into one channel or the other—this did not work for television. When you saw music on television, the instruments were not separated into left or right, but more of a mix in the middle. This may have been because early mixing decks did not have panorama controls.

The Problem With Surround Sound
Once audio technicians ventured into two channels of sound, the problems began immediately with phasing and even to the extreme of phase cancellation. With surround sound, this became even more problematic. Stereo and surround sound cracked the door open for the complete auditory experience, while the next step, immersive sound, blew the door off its hinges with creative possibilities while solving many technical issues.

For the audio producer/broadcaster, immersive sound adds an upper layer of sound over the basic surround-sound bottom layer. Technically, when you propagate the upper channels with sound, you have immersive sound.

Immersive sound is easy to produce—begin with atmospheric enhancements so the listener further believes the “being there” experience. The ambience at sports venues is fairly homogenous and omnidirectional, particularly in the upper stratus, so to capture a stable immersive sound base will require a minimum of four microphones spaced some distance apart and some distance from the source.

There is much discussion of the separation of “spaced pairs” of microphones. I happen to like a widely spaced group of microphones because I like a broad sense of dimension—my experience has been that precise placement is not absolute in sports.

With immersive sound, you do not want to over-mix the ambiance/atmosphere into the lower channel—use all of your channels to separate your sound elements and be careful not to drown out the commentators or other voices with ambiance and atmosphere.

Evolving Sounds of the Game
Immersive sound will evolve as the practitioners and producers gain knowledge and experience. There are sports that are covered with the athlete in full frame from head to toe, which lends itself to specific sounds in the upper front channels. For example with basketball, the net microphones can be placed in the upper front channels, giving a natural soundscape, because the listener does expect that the sound of the basket and net are above their head.

It is often assumed that if you cannot properly hear the sound field, you cannot properly control the sound field. This is partially true, but consider that somewhere in a quality-control or transmission room the entire immersive sound field should be verified, but for simple atmospheric enhancement visual metering is a valuable asset to the mixer.

Critical listening of a transmission mix may dictate your level of sophistication in the immersive mix. For example, basic atmospheric enhancements can be metered and monitored in a QC listening room with a soundbar to determine proper levels of sound and balance to the immersive sound encoder.

In a space without overhead speakers, you can hang a couple of overhead speakers or—depending on budgets and time—a temporary immersive mix area can be assembled for the event. Specifically, any event where “flight pack” equipment is set up, you should be able to configure speakers to accommodate Immersive Sound production.

Digital Advances
Immersive sound was not possible before digitized audio. Additionally, digitizing the audio and video signal solved the problem of transporting the broadcast signal to the consumer. The introduction of ATSC 3.0 audio standards for immersive sound with Dolby Atmos and its competitor, Fraunhofer’s MPEG-H, along with the proliferation of modestly priced encoding “black boxes” like the Liner Acoustics brands, has allowed for immersive sound to be encoded on virtually every electronic entertainment production.

Dolby Atmos and MPEG-H decode immersive sound into the home over speakers, soundbars and ear devices and either codec is capable of decoding virtually any listening configuration—not only immersive but surround and stereo, solving the decades-long concern over “the down mix.”

Entertainment formats and technologies are consumer-driven—no matter how cool the industry thinks a format is, the consumer may think otherwise. The problem for multichannel sound has always been delivering the experience to the home. The early analog “matrixes” from Dolby sounded mediocre at best, but opened ears to the possibilities of the enhanced audio experience.

Fast-forward to today and soundbars with audio decoders are the “go to” audio device for the home consumer. The consumer/listener can tell the difference with even the most basic of sound reproduction devices, and enhanced sound clearly stands out with virtually all content—drama, music, variety and sports. Finally, soundbars are significantly easier to install and set up; ask my mom.

My advice has always been, it is never going to be perfect! It only needs to be entertaining and immersive sound is entertaining.

There have been a series of significant milestones in the evolution of Olympic broadcast sound culminating in the greatest audio production of an Olympics ever. Clearly NBC has led and dominated the soundwaves for more than four decades and 2024 is the pinnacle of their persistence.

Host Broadcasters
Basically, live television sound was mono until the 1980s with the first Olympics stereo broadcast in 1988. Under the direction of sound designer Bob Dixon, NBC placed a shotgun microphone alongside the host broadcaster’s single shotgun microphone to capture XY stereo. This was no easy feat politically or technically since the Host Broadcaster Korean Broadcasting System (KBS) was only obligated to delivered mono sound to the rightsholders, including NBC.

The Host Broadcaster was traditionally the national broadcaster of the host country, but after 2008, Olympic Broadcast Services (OBS) became the permanent Host Broadcaster, fully under the direction of the International Olympic Committee.

The Host Broadcaster for the 1992 Games in Barcelona was TVE, Spain’s national broadcaster, where they produced the Opening and Closing Ceremonies in stereo, but sports were captured and produced in mono. The year 1996 was the first Olympics that the Host Broadcaster, Atlanta Olympic Broadcaster (AOB), captured and produced all events in stereo sound.

After 1996, Dixon encouraged Mike Edwards and Ken Reichel of Audio Technica to manufacture a stereo shotgun. In fact, Edwards initiated and supervised the development of three stereo microphones and two mono shotgun microphones, which were still heard on all Olympic sports and ceremonies in Paris.

Signal Management
In addition to the extensive development of suitable microphones, there was considerable work to be done with signal management and distribution. Broadcasters had to solve the problem of multichannel audio over a stereo infrastructure. Various schemes were developed to get more than two channels of sound to the home viewer/listener, but it was not until the sound was digitized that a credible surround sound was possible for distribution and transmission.

In 2006, after consulting with his audio director, Olympic Broadcast Services chief Manolo Romero determined that producing surround sound with unprocessed, discrete audio channels was the best way to satisfy the needs of all the rights holders at the Olympics—including NBC.

With 2008 came the implementation of surround sound at the Summer Olympics with the host broadcaster delivering six discrete channels of sound for the 5.1 sound format.

The sound of the 2008 Games was a significant challenge for NBC because so much of the viewing/listening audience was still listening in stereo since soundbars had only recently entered the marketplace in early 2000s.

NBC continued to develop surround sound with various Dolby analog schemes, but it was NBC’s adoption of ATSC 1.0 with Dolby AC3, along with a market full of affordable soundbars that multichannel sound would take off.

Immersive Sound
NBC polished its surround sound coverage but was persistent with the goal of true immersion. For the 2012 Games in Russia, NBC Sports Sound Designer and Olympic Supervisor, Karl Malone vigorously pursued immersive sound, beginning in Rio where NBC mixed Opening and Closing Ceremonies in immersive sound.

Pyeongchang and Tokyo followed, and immersive sound was heard in both Opening and Closing Ceremonies as well as big venues. For Beijing, all primetime coverage was produced in immersive sound. Finally in Paris, all primetime, USA Network and all Olympic sports received the immersive sonic enhancements.

By the Tokyo 2020 Summer Games, immersive sound was also captured and delivered by the permanent Host Broadcaster OBS. Nuno Duarte, the OBS sound designer and supervisor, was dealt a surprise—no audience! And the same for Beijing. The magic of sports and particularly the Olympics is seeing and hearing the arenas and stadiums full of spectators.

The games were delayed a year and everyone had time to prepare a design without spectators. The actual sound signal capture and production of immersive sound is not difficult particularly if you have a sonic layer of ambiance and atmosphere.

Malone had to make some difficult decisions about the sound design.

“Tokyo gave us the ability to focus on the details of the sports and the athletes,” he said. “The sound of the hands on the gymnastics apparatus and the creak of the wood on the parallel bars; the actual physical exertion of the athlete, the breath, the sighs, the joy all there to be captured without the masking by the crowd. We knew we were missing the passion of the crowd as much as the athletes were, but adding any fake crowd was unthinkable, even for a company like NBC, which is largely an entertainment one.”

A Milestone
The 2024 Summer Games in Paris were also a milestone as they were the first summer games where immersive sound was produced with the return of the audiences, the third dimension that was missing in Tokyo.

NBC seamlessly creates a production mix that includes “stems” of the host broadcaster’s 5.1.4 mix, plus any microphone splits, any additional crowd capture, any camera microphones, plus commentators, replays and music. Along with NBC’s branding and personal touch, it makes it look like NBC did the entire production.

“The absence of the crowd in an event of the magnitude of the greatest sporting contest in the world is almost unthinkable, and Paris is a return to what an event of this scope requires for the greatest athletes, the biggest crowds and the loudest cheers,” Malone said.

I anxiously tweaked up my Yamaha, Dolby Atmos-equipped soundbar and paid my subscription to Peacock over my Roku streamer and noticed a significant improvement with the sonic quality of the broadcasts.

The return of the spectator and the abundance of “open-air” stadiums has created a rich gumbo of sound succulence! I applaud Nuno Duarte and Karl Malone and thoroughly enjoyed listening.

With the advent of media over IP and the quest for better entertainment experiences, audio has finally taken on the prominent role in broadcast it deserves. You might even call it a next-generation audio revolution.

The industry’s embrace of immersive audio will have a tremendous impact on live broadcasts—and it’s already transforming the music, theater, movie, studio, entertainment music, studio and recording industries.

IMMERSIVE AUDIO BECOMES A REALITY

Extending surround sound, which operates in a horizontal plane, to a three-dimensional audio experience (spatial audio), immersive audio gives creatives and engineers more freedom in conveying 

emotion and more powerful tools in adding perspective to what is being watched or heard. The intent is to immerse the audience from any—if not every—direction.

Immersive audio uses ambisonic signals that capture sound in the audio field from one or several points in space, creating “audio objects” that allow for mix adaptation for different types of sound systems. Enabling manipulation of the sound field or the selection of specific streams, audio objects open the door to creation of unique experiences, both by the production team and by listeners themselves. Flexibility and choice are the key.

With immersive audio, the viewer-listener can select specific audio objects for inclusion in the version of the broadcast they see and hear. Much like an additional camera feed option, such as a “goal cam” or “referee cam,” different audio feeds can be chosen to bring the listener into different parts of a game or other live experience. Viewer-listeners can alter their spatial position and thus gain a different perspective that adds greater realism and a sense of being in the midst of the action. Additional sound might be the feed from a mic’d-up player, or communications between a Formula 1 driver and crew, or just about any other audio element that can be captured during production.

ADDED SOURCES AND DATA INCREASE COMPLEXITY

Traditional sound fields today are stereo and surround sound: 5.1 and 7.1, with left, center, right, left surround, right surround and subwoofer/low-frequency effects. Immersive audio introduces two more layers—a height layer and top layer in addition to the surround sound base layer.

Audio objects have associated metadata that describes how the audio is going to be reproduced within the sound field. Describing the position, spread, motion characteristics and other rendering information, this data gives objects the independence essential to flexible placement according to position or via a specific speaker. 

In immersive audio implementations, broadcasters will find more audio flows to contend with and manage. As the format gains a foothold in production workflows, one key challenge will be to scale services, given the much greater potential for complexity than with convention stereo and surround formats.

The number of audio sources, flows and objects is higher with immersive audio than in a standard surround sound workflow, possibly requiring more comprehensive PTP network practices to maintain timing/synchronization integrity across the workflow. Primary considerations for infrastructure will be traffic and bandwidth management, QoS treatment and proper PTP network hierarchy treatment.

Looking at the challenge broadly, one response in terms of switching infrastructure is to integrate with automation, control and orchestration platforms, as these represent the intent of the workflow by means of control. Tackling infrastructure in more specific terms, broadcasters will need to implement PTP-aware managed IP switches that can expose northbound APIs for control and provisioning.

From a networking perspective, broadcasters might want to consider bundling audio sources and objects based on categories, such as location of the source, type of sound being “collected” and specific effects to be captured. With audio flows being managed as objects that can be moved around, it’s helpful when the underlying infrastructure can aggregate data in a way that makes sense to the human brain.

To maintain logical organization of flows and data, it is useful to work with a switching infrastructure that can manage this traffic adeptly, apply groupings by means of VLANs and guarantee that the proper QoS is applied. This is especially important in workflows where other types of traffic (e.g., intercom, metadata) ride the same infrastructure.

Artel’s AoIP, managed and PTP-enabled switches provide the level of traffic control/management and PTP functionality to support immersive workflows. The boundary clock function within the company’s Quarra family of switches provides a tool for sectionalizing the PTP network to achieve better performance. (Smart sectionalizing of the audio sources can prevent PTP grandmaster overload conditions). For live events in particular, broadcasters might consider deploying their networks in terms of which switches serve which sources.

ADVANCED IP SWITCHES SIMPLIFY ADOPTION

Immersive audio is still in relatively early stages of development, and no standard configuration has yet been adopted by the industry. Broadcasters have performed trials with 5.1.2, 5.1.4 and 7.1.4, with 5.1.4 apparently offering a superior 3D experience while enabling a manageable infrastructure transition. 

Whatever format they choose to adopt, broadcasters working with advanced IP switches can be confident in their ability to address challenges that come with managing more sources and data with efficiency, and with the accuracy essential to real-time media workflows.

As we come to the end of another decade in broadcasting I think this period of time was as dynamic and influential as any for the world of audio. Not so much with new or better technology, but with a convergence of the technologies to make possible advances in multichannel audio production, delivery and consumption a reality.

THE OLYMPIC EFFECT

The pace of adopting the next generation of sound is on schedule. In 1996 we heard the full implementation of stereo sound at the Summer Olympics and in 2008 we saw 2K and heard 5.1-surround sound from Beijing. At the Summer Olympics in 2020 there will be the full rollout of 4K and 9.1 immersive sound showing a clear trend where every 12 years the broadcast industry uses the Olympics to introduce or prove technologies. There is no doubt that immersive sound is quickly finding its way to the consumer through streaming and from progressive networks around the world such as NHK in Japan, Korea’s KBS and NBC in the United States, who understand the future of broadcasting.

This past year I looked into various immersive sound applications and technologies and will continue this exploration into dimensional sound in 2020 when world broadcasters roll out the Olympics in full 4K HDR and immersive sound.

NBC is planning to mix immersive sound from a control room in Tokyo and will feature content from high viewership events such as beach volleyball and opening and closing ceremonies. Karl Malone, Audio and audio systems engineering designer for NBC Sports and Olympics said that NBC has benefitted from its experience at the 2016 and 2018 Olympics Games plus the 2019 Notre Dame football season where immersive sound was produced. I think Karl has a plan and NBC will commit the resources for a successful immersive sound production. I’ll make sure to cover it.

The future for dimensional and interactive audio is bright, but recently I read something that disturbed me: A sound manager for a large 2020 sporting event was complaining about inadequate resources to produce immersive sound. Resources or not, the requirement for immersive sound is on the table just as the requirement for surround sound was on the table in 2008—even with woefully inadequate resources and an inexperienced crew.

CREATING IMMERSIVE SOUND

After this year-long exploration of immersive sound in the pages of TV Technology I am further convinced that it is not hard to create and produce immersive sound with minimum resources and would like to offer a couple of observations:

First, to create immersive sound a mixer needs to be able to hear the dimensional sound-field. Second, the mixer must be able to adequately position the sound elements in the sound-field; and finally the mixer must be able to make and maintain an artistic balance of the elements in the dimensional sound-field.

In the April issue of TV Technology I visited an OB van with less than perfect but adequate immersive monitoring. I can only say that NHRA puts out a credible 5.1.4 immersive sound production as well as 5.1 surround from one OB control room.

An alternative workflow would change the location for mixing and monitoring immersive sound. For more than a decade, FIFA World Cup and NBC have been sending mono and stereo stems from the venues back to master control to be mixed and I can clearly see the further trend to centralized control rooms for multiple remote productions.

There also seems to be some schools of thought that you need ambisonic microphones to create immersive sound. I spent 2019 recording side-by-side a 1st, 3rd and 4th Order ambisonics microphone along with 10 correlated and non-correlated spot microphones. I’ve come to the conclusion that ambisonic microphones sound great, but are not essential to creating immersive sound.

Immersive sound for sports and live entertainment is a subjective balance between venue ambiance and atmo-sphere, as well as relevant event-specific sound. With any sport (event) minimum immersive sound production can be as simple as injecting additional ambiance and atmosphere into the height speakers. To me, American and European football (soccer) are the definitive example of overhead atmospheric enhancements and this method does not require any localization. 3D panning is helpful for precise localization, but bottom line—convincing immersive sound for most sports does not require precise localization.

Some sound above the viewer/listener will usually create a sense of aural space for the 2D picture, but sound designers are wrestling with the concept of what sounds should be heard above the viewer when there is no obvious reason. Sure, we can put some sound up in the height channels, but how long is more crowd sound going to be a compelling reason for immersive sound.

SUCCESSFUL IMMERSIVE SOUND

Immersive sound production will develop and evolve with creative sound design and imagination. The picture is still two-dimensional and sound could become a production differentiator where “made for TV” sound such as NHRA will become a viewer draw.

There is no doubt that field sports such as football have no sports-relevant sound in the vertical axis, only ambiance and atmosphere, and artificial embellishments are probably not appropriate. But consider that perhaps enhanced ambiance is enough of an embellishment for football fans—immersive sound does not need to be over the top to be effective.

Hoping for interesting and successful immersive sound is not going to advance our audio agenda. NHK in Japan has spent almost two decades preparing for immersive sound. Dreaming, theorizing, research, testing, evaluation, planning, preparing new sound designs and documents have contributed to and facilitated a successful implementation of immersive sound within the Japanese broadcaster. NHK will produce the entire 2020 Olympic Games in 22.2 immersive sound.

Immersive sound will be proven one step at a time. Immersive sound is not new, but the concept of immersive sound is evolving. Remember, I said NBC would produce 9.1 immersive sound and NHK would produce 22.2; both formats claim to be immersive sound. What is the difference on a soundbar?

Advance audio designs are eminent but will be built on proven successes. What is the channel configuration for immersive sound? Is it 5.1.2, 5.1.4 or 22.2? I do not know. I am currently testing sound schemes that concentrate the listener’s attention forward and uses the height element to draw the focus beyond the normal left and right peripherally. This is an interesting concept for immersive sound and seems to work well with soundbars.

The migration to immersive sound must be planned, methodical and most importantly successful. Immersive sound must be marketed to the consumer and easy to install with minimum wires. Stereo and surround sound had growing pains, but I could not imagine going back to mono sound or black-and-white TV.

Immersive sound has a promising future and I look forward to covering it in the pages of TV Technology.

Dennis Baxter has spent more than 35 years in live broadcasting contributing to hundreds of live events including sound design for nine Olympic Games. He has earned multiple Emmy awards and is the author of “A Practical Guide to Television Sound Engineering,” published in both English and Chinese. He is currently working on a book about immersive sound practices and production. He can be reached atdbaxter@dennisbaxtersound.comor atwww.dennisbaxtersound.com.

After talking to many of my colleagues, I will confidently say that if a sound mixer can reasonably hear immersive sound, they can mix it. The question is: How do we get there?

There are immersive sound mixing stages and studios in Hollywood and New York, but the number of mobile audio mixing facilities are few. Solutions are evolving; but it appears that adding more speakers and modifying existing OBs is the path forward for sports immersive sound. Generally, only the audio mix positions will need to be fitted for 5.1.4 immersive sound monitoring.

POSITIONING SPEAKERS

In my April article in TV Technology, I mentioned that Game Creek and NEP are OB providers that have modified existing OB vans for immersive sound mixing. For the National Hot Rod Association (NHRA) Drag Racing, Game Creek installed similar speaker setups in both mix rooms for A1 Josh Daniels and effects mixer Rusty Roark to listen to. I was told that the mix rooms were small and there had to be compromises with the positioning of the speakers.

Before my visit, I asked Mike Rokosa with NHRA to send me a 5.1.4 production clip to preview on my 11.1 Genelec system. In the studio, I was impressed with the movement of the sound in the horizontal and vertical soundfield and even more amazed after I personally spent a weekend at the drag races. I sat in the A1 mix position and listened to a couple of races.

I could clearly hear the intricacies of the immersive effect mix in the OB van. I admit I was surprised at the positioning of the height speakers, but did not let my eyes prejudice what my ears heard, particularly since I had already auditioned the mix in an accurate monitoring environment.

Most OB vans have limited space to mount more speakers, much less accurately align and calibrate them. According to Thomas Lund, chief technologist at Genelec Speakers, compact speaker designs are available, and Genelec’s 8331 systems had been installed in applications with a listening distance of as little as 70 cm, which still delivered an accurate representation of the soundfield to the mixer.

I also talked to Mike Babbitt, an applications engineer with Dolby, about tuning the speakers in the Game Creek OB mix rooms. Babbitt worked with NHRA on the implementation of immersive sound.

“All speakers [not the sub] were set to reproduce the same SPL at the listening position using ‘pink noise,’” he said. “For the sub, we try to set it with 10 dB in-band gain compared to the center channel, which approximates to about a 3 dB SPL bump using an SPL meter.” There was no time for alignment of the speakers.

Another compromising factor is that the immersive 5.1.4 mix is being accomplished on concentric 5.1 busses. Although I am a big advocate of true XYZ panning, all I can say is that the NHRA sound model is very creative, clever and it works for their sport. (Watch for my interviews with Daniels and Roark detailing the audio mapping and mixing that makes this an artistically creative mix.)

ALTERNATIVE MIXING

As immersive sound production develops, we should consider alternative mixing solutions for live sports and entertainment. Mixing the immersive sound (Surround Sound) from a different location has been successfully accomplished for several World Cup productions. Master control should be monitoring immersive sound and could easily composite the audio stems from the OB van and build an immersive mix in a centralized control room. This is clearly a solution for many field-type sports such as baseball, football and soccer.

One final consideration: I suggest researching virtual/programmable soundfield reproduction for the audio mix room. Beam steering of microphones and speakers to create virtual soundfield patterns is a developed and well-understood technology. As critical listening space becomes a necessity, perhaps an alternative approach to monitoring in the OB van should be assessed.

There will be a variety of immersive sound productions that requires a range of immersive sound monitoring. Critical listening for localization may be far more important in film, but when the height channels are a wash of distant crowd and PA, then I would suggest that, for many sports (not all), critical listening is more of a judgment about spatialization of the sound elements than localization. Also, consider that the soundbar is the probable method of delivery for immersive sound and that precise localization from up-firing and side-firing speaker configurations is not possible in most home applications.

It seems as though the OB van companies are finding solutions. By modifying existing mix space, this will facilitate the conversion to useable immersive sound mixing spaces. Perhaps there should be a rating system for mixing spaces so that you can evaluate an OB van beyond how many cameras or monitors there are for production.

I think many would acknowledge that the migration to immersive sound will be a layer and a step at a time. NHRA is testimony to the fact that convincing immersive sound can be produced today.

Immersive sound technology is getting ready. Are you?

Dennis Baxter has spent over 35 years in live broadcasting contributing to hundreds of live events including sound design for nine Olympic Games. He is the author of “A Practical Guide to Television Sound Engineering,” published in both English and Chinese, and is currently working on a book about immersive sound practices and production.

He can be reached atdbaxter@dennisbaxtersound.comor atwww.dennisbaxtersound.com.