DriftSE — Speech Enhancement Based on Drifting Models

Speech Enhancement Based on Drifting Models (Interspeech 2026, Oral Presentation)

Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan, Rasmus Kongsgaard Olsson

🔗 Project Website  |  📄 arXiv Paper  |  💻 GitHub Code|  🤗 Hugging Face Space

---

✨ Key Highlights

• Novel Generative Paradigm — Formulates speech enhancement as a distributional equilibrium problem, eliminating the need for iterative denoising or trajectory-based sampling.
• Native One-Step Inference — Achieves single-step (1 NFE) enhancement by evolving the pushforward distribution of a mapping function to directly match the clean speech distribution via a Drifting Field.
• Semantic Latent Drifting — Operates in a hierarchical self-supervised speech latent space (HuBERT, WavLM, DistilHuBERT), providing rich and stable training signals that capture both acoustic and phonetic structure.
• Unpaired Learning — Natively supports training on fully unpaired noisy/clean speech data, enabling cross-dataset and cross-gender generalization without paired supervision.
• State-of-the-Art Generalization — Achieves state-of-the-art WV-MOS and SCOREQ on the DNS Challenge 2020 blind test set, outperforming multi-step diffusion and consistency-based baselines.

---

📦 Repository Contents

This Hugging Face repository hosts pre-trained checkpoints and enhanced audio outputs for DriftSE.

LIANGXU123/DriftSE/
├── logs/                                          # Pre-trained model checkpoints
│   ├── distillhubert_three_layers_with_z/
│   │   └── last.ckpt                              # DriftSE (DistilHuBERT) — conditional generator
│   └── distillhubert_three_layers_pesq_sisdr_ccmse_with_z/
│       └── last.ckpt                              # DriftSE† (DistilHuBERT) — with auxiliary losses
└── out/                                           # Enhanced audio outputs
    ├── distillhubert_three_layers_with_z/          # Enhanced VB-DMD test set (DriftSE)
    └── distillhubert_three_layers_pesq_sisdr_ccmse_with_z/  # Enhanced VB-DMD test set (DriftSE†)

---

📊 Performance Benchmark

VoiceBank-DEMAND (VB-DMD) — In-Domain Evaluation

† Jointly trained with auxiliary PESQ, SI-SDR, and CCMSE losses.

DNS Challenge 2020 Blind Test Set — Real-World Generalization

---

🚀 Quick Start

1. Install Dependencies

git clone https://github.com/liangxu123/driftse.git
cd driftse
pip install -r requirements.txt

2. Download Checkpoint

from huggingface_hub import hf_hub_download

# DriftSE (DistilHuBERT) — conditional generator
ckpt_path = hf_hub_download(
    "LIANGXU123/DriftSE",
    "logs/distillhubert_three_layers_with_z/last.ckpt"
)

# DriftSE† (DistilHuBERT) — with auxiliary PESQ/SI-SDR/CCMSE losses
ckpt_path_aux = hf_hub_download(
    "LIANGXU123/DriftSE",
    "logs/distillhubert_three_layers_pesq_sisdr_ccmse_with_z/last.ckpt"
)

Or via CLI:

# Download the full repository
huggingface-cli download LIANGXU123/DriftSE --local-dir ./DriftSE_hf

3. Run Enhancement

bash ./test.sh <GPU_ID> [CONFIG_PATH]

# Example: default config (DistilHuBERT, conditional generator)
bash ./test.sh 0

# Example: specific config
bash ./test.sh 0 ./config/with_z/v2_drift2_distillhubert_three_layers.json

The evaluation pipeline runs two phases:

1. Enhancement — generates enhanced audio via enhancement.py
2. Objective Metrics — computes PESQ, ESTOI, SI-SDR via calc_metrics.py

---

🏗️ Model Architecture

Two Formulations

• Conditional Generator (with_z/) — Stochastic mapping f_θ(ε, y) from Gaussian noise conditioned on noisy speech, optimized for perceptual quality (DNSMOS, SCOREQ).
• Direct Mapping (no_z/) — Deterministic mapping f_θ(y) from noisy to clean speech, with σ=0 for highest PESQ/SI-SDR fidelity.

---

🤗 SSL Encoder Checkpoints

DriftSE requires frozen self-supervised speech encoders to compute the latent drifting field during training. These are not needed for inference with the pre-trained checkpoints above.

Download: Google Drive — latent_ckpt/

latent_ckpt/
├── wavlm-large-local/       # WavLM-Large (1024-d, 24 layers)
├── hubert-large-local/      # HuBERT-Large (1024-d, 24 layers)
└── distilhubert-local/      # DistilHuBERT (768-d, 2 layers)

---

🔧 Training

To train DriftSE from scratch:

bash ./train.sh <GPU_ID> [CONFIG_PATH]

# Example: default DistilHuBERT config
bash ./train.sh 0

# Example: with auxiliary losses (PESQ + SI-SDR + CCMSE)
bash ./train.sh 0 ./config/with_z/v2_drift2_distillhubert_three_layers_pesq_sisdr_ccmse.json

Training uses dynamic mixing: 10,802 clean VoiceBank utterances are mixed on-the-fly with 18 DEMAND noise types at SNRs sampled from {0, 5, 10, 15} dB.

---

📝 Citation

If you find DriftSE useful in your research, please cite:

@inproceedings{xu26d_interspeech,
  title     = {{Speech Enhancement Based on Drifting Models}},
  author    = {Liang Xu and Diego Caviedes-Nozal and W. Bastiaan Kleijn and Longfei Felix Yan and Rasmus Kongsgaard Olsson},
  year      = {2026},
  booktitle = {{Interspeech 2026}},
  pages     = {1369--1374},
  doi       = {10.21437/Interspeech.2026-833},
  issn      = {2958-1796},
}

@article{xu2026driftse,
  author  = {Xu, Liang and Caviedes-Nozal, Diego and Kleijn, W. Bastiaan and Yan, Longfei Felix and Olsson, Rasmus Kongsgaard},
  title   = {{DriftSE: Speech Enhancement with Generative Drifting}},
  journal = {arXiv preprint arXiv:2609.12252},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.12252}
}

---

📄 License

This project is licensed under the MIT License.

---

🙏 Acknowledgments

We thank the authors of SGMSE+ and Drifting Models for the foundational work that inspired this codebase.