Immersive Audio: "Everybody Is Very Sceptical About It, Until You Play Them Something"

In just a few short years, Dolby Atmos playback has evolved from select high-end theaters, all the way to popular consumer electronics, home cinema systems, video games, and, soon, even cars. Headliner recently chaired a panel discussion with four experts in the field of immersive audio to reflect on how far the various formats have come, and to look at trends across the music and entertainment industries.

On the panel were Dave Way, a four time Grammy award-winning producer and recording engineer who has worked with Michael Jackson, Spice Girls, Beck, Eric Clapton, Neil Young, Paul McCartney, Mick Jagger and Gwen Stefani, and Grammy-nominated recording engineer Steve Genewick, whose credits include working on records for LeAnn Rimes, Bob Dylan, Mary J Blige and Elton John.

Joining them were two members of the Abbey Road Spatial Audio Forum, Ana Monte, a sound designer with a focus on immersive audio and virtual reality, and Anastasia Devana, an expert in immersive audio.

How does Dolby Atmos heighten the listening experience and immerse the listener?

SG: I like to say it's surround sound that actually works. It really opened up huge possibilities in what we could do with music and how we could use the system to literally immerse the listener inside of the song. It's a more compelling way to suck you in, for lack of a better term.

AM: The coolest thing about immersive audio is how you change the POV. You're not outside anymore; you're in the story. You're in the music, if you will.

Steve was tasked with up-mixing a lot of UMG and Capitol’s back catalogue into an immersive format. What were the challenges?

S.G: UMG came to us at Capitol Studios and they said, ‘You guys have got to build a Dolby Atmos room’. And we said, ‘why would we want to do that?’ They said, ‘just build it, we're gonna mix music!’ We took our studio CR/mix room and retrofitted it for Atmos, and it turned out great. Basically, I had to just figure it out.

I had mixed in 5.1 and I've done movie scores, so surround formats were not completely foreign to me. Luckily, Atmos was integrated into Pro Tools very recently before that, but then it was just me sitting in a room trying to figure out how to make this work and how to make sure that I was mixing a record, and not a Dolby Atmos demo.

I had to figure out how to make reverb work and how to get what we do as music mixers into this immersive format.

What are the learning curves when adapting to an immersive format?

DW: There's a pretty steep learning curve, technically with the renderer and the communication between Pro Tools and getting your room set up. The thing about Atmos is you can do whatever you want with it; it really is the Wild West as far as where you want to put things!

Often you're experimenting with where the vocal is, because that's usually the star of the show. Having it up front, oftentimes is just boring, and you're trying to get a little more of a novel thing by using the space to approach it. That's usually the first place to go, and then the music finds its place after that.

What about immersive / spatial audio in interactive game engines?

AM: I came from a traditional sound design background for films, and it's definitely a completely different approach. Your POV has changed: now you're in the story, you're not watching things from out there, so you need to be careful to not overload the viewer. As sound designers, we love to layer, layer, layer!

That's not the case with VR because you're trying to make something that looks and sounds realistic, and sometimes that cinematic sound design that we like doesn't work that well with VR, because you want something that's more real. So the approach is very different.

The way that you tell a story in VR is still being developed, like how do we tell a story where the viewer is in the story, and is part of the story? How do I make my sound design complement the story and not distract the viewer? Sound design is adapting towards that.

Do you think that more artists will embrace creating music in immersive audio formats going forward? And will audiences embrace it?

SG: I have yet to have somebody come in and go, ‘I hate this’. Everybody is very sceptical about it until you play them something. After they hear something, they all love it. Nine times out of 10 it's, ‘how do I get my new album this way?’

AD: I used to work at a tech company that makes an augmented reality hardware platform, and I was experimenting with object-based immersive audio using a game engine. It's like going from mono to stereo, but then you go from stereo to this full spatial audio.

We had this quite famous music producer come in to see the technology. I said, ‘check this out, it’s like going from mono to stereo’, and he was like, ‘meh; people listen to music on phone speakers – they're not gonna care’. He crushed my dreams of spatial music! I did not expect the music producer to not get it.

DW: I have to say, across the board, for me every reaction has been, ‘wow’. I can see the goosebumps rising on their arms and they turn around and they smile.

Everybody's like, ‘how can I get this? What do I have to do? How do I do this at my house? How do I start mixing in Atmos? How much does it cost?’ Just in the seven or eight months that I've been doing this, the momentum and the awareness has grown measurably, and I get more and more excited about it every month. I had a client say ‘it's like you're swimming and the water is music’.

Tell us about the Abbey Road Spatial Audio Forum.

AM: It was founded in 2017 by Mirek Stiles, who is Abbey Road’s head of audio products. He decided to bring together a bunch of audio nerds from completely different areas: artists, producers, engineers, people from music, VR and film, from the academic world, and get us all talking to each other, which apparently doesn't happen very often!

Meaning the researchers talking to the people that are actually doing the content, and vice versa. We try to meet regularly (when there isn't a worldwide pandemic), and exchange ideas – from research, all the way to practical stuff that we've experimented with to try to find new, creative approaches to using immersive audio.

Tell us about the world of ambisonics and object-based ‘mixing’ within game engines.

AM: With ambisonics, you're not dealing with object-based or with channel-based audio – you're trying to reproduce a sound sphere. That's why it's been used for VR, because in VR and 360 degree videos you want a sound sphere around you so you're able to look in every direction, and have head tracking so you can rotate the sound field.

There's a lot of flexibility. I personally don't think it's a good format for cinema – I think Dolby Atmos is definitely better, being object-based and channel-based. But for 360 degrees and for VR, it's a perfect format. You want to be in the middle of that sphere and have everything surrounding you, and you want the sound field to be able to rotate.

How have you been using Focusrite gear in combination with Dante networking for your immersive audio work?

SG: At Capitol we use a lot of Focusrite stuff because we're moving between three computers, so we're using Focusrite hardware and Dante to move audio between those three computers constantly.

I did a video series where we put bands on the roof of the Capitol building, and we had Focusrite boxes up there and we would use RedNet to pull the audio down into the studio so I could mix it.

When I put an Atmos studio in my house, I naturally went to Focusrite. I ended up with a Red 16Line and the RedNet R1, and I couldn't be happier. It's so easy to set up and use, and it sounds great. I can easily switch between stereo and Atmos. I was amazed at how good it actually sounded.

DW: I’ve got a Red 16Line, a couple of RedNet A16Rs, a RedNet HD32R for more Dante, plus a RedNet R1 controller. It's seamless, and the RedNet software is just fantastic – everything's right in front of me and I don't think about it at all.

I'm easily going back and forth between stereo, 5.1 and Atmos, and I also have an Adobe receiver where I'm streaming Tidal, and soon, Apple Music. It sounds great – that's always number one. I don't want to use anything that's convenient, but doesn't deliver on the sound.

What trends are you seeing in immersive audio across the music and entertainment industries?

AM: This is something we talked about in the last forum: how are we going to be mixing music in the future, especially with these interactive, augmented and mixed reality experiences? In the future, or actually, it's already happening, we are thinking, ‘how do we control what the viewer or what the person is experiencing?’

AD: Headphones like AirPods Pro and AirPods Max do head tracking like spatial audio where it has to react to your head movements. These will render anything in Dolby Atmos correctly. I've heard it! So I think that basically it’s getting people to experience it and to realise what spatial audio sounds like.

AM: Exactly. We have a lot of things being consumed with headphones. For me, it's super interesting, not only from a music aspect, but obviously for movies that people are consuming on their way to work on their cell phones.

They're listening to things on their tablets. So from a movie perspective, it’s really important that we have good binaural mixes for video also.

DW: I'm continually excited about what the possibilities are with this kind of music. Some are meant to be more natural, for instance I just did the UEFA opening ceremony for the soccer tournament, and it was Marshmello who did a medley, and the visuals for it were absolutely surreal.

You’ve got this guy with a marshmallow for a head and it's all CGI stuff all over the world, and oceans are in the middle of the stadium, so it really lends itself to the surreal stuff, even though we're trying to give you the sense that it's ‘grounded’ in the stadium. That kind of stuff is fun – every project is different.

SG: I like the fact that it's coming to the forefront and seems to be on everybody's radar. The more people talk about it, the more it's gonna stick around. Hopefully the consumer stuff catches up fairly quickly, which it looks like it is. I'd like to be able to get this out to more people.

This is the first time where the creators have been ahead of the consumers. It's not the consumer manufacturers going, ‘hey, we have this great toy; now you guys figure out how to use it’. It's us saying, ‘okay, here's the music; we're mixing the video games, we're doing the movies – you figure out how to play it back’.

I like it being driven that way: the content driving the technology.