Blend real and virtual sounds seamlessly by creating binaural illusions of acoustic sources.

Concept

Augmented and mixed reality systems overlay virtual sound onto the physical world. For the illusion to hold, virtual sources must be acoustically indistinguishable from real ones — a challenge that demands accurate binaural rendering, precise spatial reproduction, and an understanding of human perception.

Our research investigates when and why listeners accept virtual sounds as real, developing evaluation paradigms and rendering techniques that push the boundaries of auditory plausibility.

Transfer-Plausibility

We developed the concept of transfer-plausibility[1][2]: a rigorous framework for evaluating whether virtual sources are accepted as real when both real and virtual sounds coexist. This goes beyond traditional authenticity testing and captures the perceptual demands unique to AR/MR scenarios. In a transfer-plausibility test, listeners try to identify the virtual source among several real ones.

Binaural Rendering for 6 Degrees of Freedom

Rendering spatial audio for listeners who can freely move and rotate in a space requires processing recorded Ambisonics sound fields with distance and position information[3][4]. Our work addresses source distance modeling and listener navigation through measured sound fields, enabling experiences like Inside the Quartet — an immersive experience placing the listener inside a string quartet[5].

Code: SPARTA 6DoFconv — plugin for six-degrees-of-freedom convolution with spatial room impulse responses.

Code: SRIR Interpolation Toolkit — perceptually informed interpolation of spatial room impulse responses between measurement positions.

Latency & Perceptual Thresholds

Low-latency processing is critical for maintaining the auditory illusion in real-time AR. We developed a method for measuring the motion-to-sound latency of virtual acoustic rendering systems[6], so that the latency of a complete head-tracked system can be verified in practice.

Code: Latency Analyzer — tools for measuring binaural rendering latency.

Head-Worn Device Transparency

Wearing headphones or AR glasses disrupts the perception of real sounds. We developed methods for predicting perceptual transparency of head-worn devices[7], informing the design of passthrough processing that preserves natural listening.

Audiovisual Congruence

How do visual cues interact with spatial audio? We studied whether loudspeaker models or human avatars in VR affect localization performance[8], revealing the interplay between visual representation and spatial hearing accuracy.

Room Acoustic Memory

Can listeners remember and compare the acoustic character of spaces? Our experiments[9] investigate how accurately listeners retain room acoustic impressions, informing how quickly AR systems must adapt when transitioning between environments.

Experiences

Inside the Quartet — immersive spatial audio placing the listener inside a string quartet, demonstrating high-quality binaural rendering for musical performance[5].
Space Walk — a navigable virtual planetarium for the Oculus Quest with spatialized music[10], combining stereophonic and immersive sound spatialization.

References

Nils Meyer-Kahlen, Thomas McKenzie, Sebastian J. Schlecht et al. (2022). Transfer-Plausibility of Binaural Rendering with Different Real-World References. DAGA - Jahrestagung Für Akustik.
Nils Meyer-Kahlen, Sebastian J Schlecht, Sebastià Amengual Garí et al. (2024). Testing Auditory Illusions in Augmented Reality: Plausibility, Transfer-Plausibility, and Authenticity. Journal of the Audio Engineering Society.
Axel Plinge, Sebastian J Schlecht, Oliver Thiergart et al. (2018). Six-Degrees-of-Freedom Binaural Audio Reproduction of First-Order Ambisonics with Distance Information. AES International Conference on Audio for Virtual and Augmented Reality (AVAR).
Olli S Rummukainen, Sebastian J Schlecht, T Robotham et al. (2019). Perceptual Study of Near-Field Binaural Audio Rendering in Six-Degrees-of-Freedom Virtual Reality. IEEE Virtual Reality (VR).
Nils Meyer-Kahlen, Petra Piiroinen, Gautam Vishwanath et al. (2023). Inside The Quartet - A First-Person Virtual Reality String Quartet Production (Non-Peer-Reviewed). 154th Convention of the Audio Engineering Society.
Nils Meyer-Kahlen, Miranda Kastemaa, Sebastian J. Schlecht et al. (2023). Measuring Motion-to-Sound Latency in Virtual Acoustic Rendering Systems. Journal of the Audio Engineering Society.
Pedro Lladó, Thomas Mckenzie, Nils Meyer-Kahlen et al. (2022). Predicting Perceptual Transparency of Head-Worn Devices. Journal of the Audio Engineering Society.
Anja Hofmann, Nils Meyer-Kahlen, Sebastian J Schlecht et al. (2024). Audiovisual Congruence and Localization Performance in Virtual Reality: 3D Loudspeaker Model vs. Human Avatar. Journal of the Audio Engineering Society.
Madalina Nastasa, Nils Meyer-Kahlen & Sebastian J. Schlecht (2021). Assessing Room Acoustic Memory Using a Yes/No and a 2-AFC Paradigm. Nordic Sound and Music Conference.
Andrea Mancianti, Sebastian J. Schlecht, Vesa Välimäki et al. (2021). Space Walk - Visiting the Solar System through an Immersive Sonic Journey in VR. Nordic Sound and Music Conference.