What’s Missing from Virtual Reality? Immersive 3D Soundscapes

When you imagine virtual reality, chances are you’ve got visuals in mind. Or maybe you fantasize about a virtual sense of touch. Another key component? Immersive audio.

Tom Simonite of MIT Technology Review recently got a demo of some pretty impressive 3D sound in Microsoft’s Silicon Valley lab. After using scanning technology Kinect to make a digital 3D model of his head and shoulders, the team gave Simonite a pair of special sensor-laden headphones plugged into some specialized software.

The result was more than simple surround sound.

Using the 3D model of his head, motion sensors in the headphones, and a camera to track Simonite’s movement, the system precisely adjusted sounds in the headphones to make it seem like they were coming from specific (previously silent) points in space—a voice on a cardboard radio, music from a speaker, and chirps from a stuffed bird.

You may have experienced a simple version of this effect when listening to music that pans left and right on headphones. Panning makes it sound like music is emanating from various points in between your ears.

But instead of me describing it to you, check out this early (but impressive) demo of positional sound for Oculus. (Best with headphones.)

Currently, positional 3D audio in video games is informed by average physiological models of the head. But the shape and position of our ears and the anatomy of our head changes how sound reaches our ear canals. Microsoft researchers, Ivan Tashev and David Johnston, say customization makes positional audio far more accurate.

The pair used 250 physiological profiles of people's heads and ears to write their software. Added to a simple 3D scan, the software can accurately tailor 3D sound to trick the ears into thinking it originates in very specific locations out in space.

“Essentially we can predict how you will hear from the way you look,” Tashev told MIT Technology Review. “We work out the physical process of sound going around your head and reaching your ears.” 

But why is Microsoft working on 3D sound at all? Presumably for the same reason they developed Kinect—for their gaming business. If you’ve got a gaming platform, chances are you’re anticipating the imminent arrival of virtual reality.

Grateful Dead drummer, Mickey Hart, experiments with the Oculus Rift.

Virtual reality has long been a dream of gamers and technologists. What’s changed?

Last year, Oculus made waves when they showed off their head-mounted virtual reality Rift device. Rift is immersive virtual reality, but more than that, it’s affordable immersive virtual reality. The firm, since acquired by Facebook, is aiming for a $300 consumer ready virtual reality gaming system later this year or early next.

Mobile computing has driven a decade of miniaturization and declining prices in sensors, processors, and screens. Oculus uses cheap but super-precise motion sensors to track head position, powerful processors to manipulate visuals to give the illusion of immersive 3D, and high definition screens to deliver quality imagery.

Microsoft’s 3D audio currently uses motion sensors embedded in the headphones and a camera to detect a user’s head position—but according to the MIT Technology Review article, sensors like those in the Rift would also work. The system would combine position data with the physiological profile of a user taken in the beginning using Kinect.

Positional audio is not a new field—for example, check out this 2007 YouTube video of QSound's Virtual Barbershop—but it is rapidly improving thanks to more widely available and affordable sensors.

Microsoft hopes to create the necessary physiological profile for 3D sound filters simple enough users can make one at home using naught but Kinect. Though at-home solutions might not match the highest quality lab-created filters, they would be good enough to greatly increase the sense of immersion in virtual worlds.

And, of course, Microsoft isn’t alone. The firm’s gaming rival and maker of the Playstation video game platform, Sony, earlier this year unveiled their virtual reality venture, Project Morpheus, and likewise hinted they are working on positional 3D audio.

Richard Marks, senior director of research and development at Sony Computer Entertainment America, told Polygon that along with visuals, his firm views sound as an equally important component to the virtual experience. According to Marks, Sony is working to create positional audio that adapts to a players' head orientation "creating a highly realistic audio environment within an immersive 360-degree virtual world."

No doubt Sony and Microsoft will be joined by independent developers. Even as Oculus has shown they’ve increasingly got virtual reality visuals nailed, there’s plenty of room for work on other tools and peripherals—specialized treadmills, body tracking systems, 3D audio, more intuitive interfaces—to better transport our senses into the virtual.

Image Credit: Janus Sandsgaard/Flickr; Christopher Michel/Flickr