Vocal Effects Style Transfer Demo

This is a demo of the paper Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior, published at WASPAA 2025. In this demo, you can upload a raw vocal audio file (in mono) and a reference vocal mix (in stereo). This system will apply vocal effects to the raw vocal such that the processed audio matches the style of the reference mix.

The effects is the same as in DiffVox, which consists of series of EQ, compressor, delay, and reverb. We offer four different methods for style transfer:

  1. Mean: Use the mean of the chosen vocal preset dataset as the effect parameters.
  2. Nearest Neighbour: Find the closest vocal preset in the chosen dataset based on chosen embedding features, and use its effect parameters.
  3. ST-ITO: Our proposed method that perform inference-time optimisation in the chosen embedding space, regularised by a Gaussian prior learned from the chosen vocal preset dataset.
  4. Regression: A pre-trained regression model that directly predicts effect parameters from the reference mix.

Datasets

  • Internal: Derived from a proprietary dataset in Sony.
  • MedleyDB: Derived from solo vocal tracks from the MedleyDB v1/v2 dataset.

Embedding Models

  • AFx-Rep: A self-supervised audio representation model trained on random audio effects.
  • MFCC: Mel-frequency cepstral coefficients.
  • MIR Features: A set of common features in MIR like RMS, spectral centroid, spectral flatness, etc.

Note: To upload your own audio, click X on the top right corner of the input audio block.

Style Transfer Method
0 1

Control Parameters

Prior Distribution (Dataset)
This parameter has no effect when using the ST-ITO and Regression methods.
Embedding Model
This parameter has no effect when using the Mean and Regression methods.
Use real-time delay and reverb effects instead of approximated ones.
This option has no effect when using the Mean and Regression methods.

Parameters for ST-ITO Method

1 2000
Prior Weight
Weight of the prior distribution in the loss function. A higher value means the model will try to stay closer to the prior distribution.
Optimiser
Learning Rate

Effect Parameters Visualisation

Effect Settings JSON

{
}