Recover hidden audio signals from degraded observations with flexible generative models.

Concept

Inverse problems concern the estimation of an unobserved signal from degraded observations produced by a forward process. They are inherently challenging because the inverse mapping is often ill-posed: solutions may be non-unique, unstable, or may not exist.

In audio, inverse problems arise in applications including stem separation, dereverberation, declipping, restoration, and timbre transfer.

Generative Models as Audio Priors

Generative approaches, particularly diffusion models and flow matching, provide a probabilistic, data-driven framework for solving audio inverse problems. Instead of explicitly inverting the degradation process, these methods approximate the distribution of possible latent signals conditioned on the observation.

In other words, the model infers plausible underlying audio that is consistent with both the observed signal and a learned distribution of natural audio. This sampling-based approach makes it possible to reconstruct perceptually convincing solutions even when the observation alone does not determine a unique answer.

One Model, Many Restoration Tasks

This formulation does not require a separate inverse model for every degradation. A single pretrained generative audio model can therefore be applied across different restoration tasks, including cases where the specific degradation is not known at inference time.

This task-agnostic perspective brings stem separation, dereverberation, declipping, and timbre transfer into a shared reconstruction framework.

Zero-Shot Timbre Transfer

The featured figure illustrates zero-shot oboe-to-piano timbre transfer using a diffusion model trained on piano recordings[1]. Starting from the oboe spectrogram, the diffusion process progressively reconstructs an output that retains the musical content while following the learned spectral characteristics of a piano.

Demo: Audio To Audio Via Diffusion Warm Initialization — listening examples for timbre transfer, MIDI-to-real synthesis, audio enhancement, declipping, and source separation.

RIRFlow: Training-Free Room Impulse Response Restoration

RIRFlow applies flow matching to room impulse response inverse problems without requiring a learned generative prior[2]. It models the exponentially decaying variance of a room impulse response as a Gaussian process, yielding a closed-form Wiener denoiser that remains interpretable through physical parameters such as reverberation time.

The analytic denoiser guides posterior sampling and measurement refinement within the flow-based solver. This framework supports denoising, deconvolution, inpainting, and declipping, including nonlinear and non-Gaussian observation models.

References