RMVPE — ONNX
RMVPE (Wei et al., RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music, Interspeech 2023) as shipped by the RVC project (lj1995/VoiceConversionWebUI, rmvpe.pt). A deep U-Net + BiGRU on a 128-band log-mel; robust to noise and accompaniment.
Exported to ONNX with the model's own front end inside the graph: feed 16 kHz mono audio, get the model's own output — no torch, librosa or reimplemented STFT/mel needed, and nothing that can quietly differ from the original. Every export was checked frame by frame against the original PyTorch pipeline (below), and measured on PTDB-TUG speech against the other exports in this family and against pyin.
| file | model | parameters | size |
|---|---|---|---|
rmvpe.onnx |
RMVPE | 90.5 M | 185.8 MB |
Same family: rmvpe-onnx · fcpe-onnx · crepe-onnx · pesto-onnx.
Use
# pip install onnxruntime numpy soundfile soxr huggingface_hub
import soundfile as sf, soxr
from huggingface_hub import hf_hub_download
import importlib.util
repo = "Scicom-intl/rmvpe-onnx"
spec = importlib.util.spec_from_file_location("pitch_onnx", hf_hub_download(repo, "pitch_onnx.py"))
pitch_onnx = importlib.util.module_from_spec(spec); spec.loader.exec_module(pitch_onnx)
model = pitch_onnx.PitchModel(hf_hub_download(repo, "rmvpe.onnx")) # threshold= to override voicing
y, sr = sf.read("speech.wav", dtype="float32", always_2d=True)
y = soxr.resample(y.mean(1), sr, 16000)
f0, voiced, confidence = model(y) # one per 10 ms frame (frame i at i * 10 ms; FCPE + 5 ms); f0 = 0 unvoiced
pitch_onnx.py is numpy + onnxruntime only: it cuts CREPE's frames, decodes the 360-bin salience
the way the models ship (salience-weighted mean of the 9 bins around the peak; CREPE/RMVPE bins are
20 cents from 32.70 Hz, FCPE's are evenly spaced 32.7–1975.5 Hz) and applies the voicing threshold.
Details
RMVPE
- Input:
audiofloat32 [1, N] — 16 kHz mono - Output:
saliencefloat32 [1, T, 360], T = N // 160 + 1 frames, frame i centred at i × 10 ms. - Front end: RVC's mel (centred reflect STFT, n_fft 1024, hop 160, 128 HTK mels 30–8000 Hz, log clamp 1e-5) and the U-Net's pad to a multiple of 32 frames are inside the graph.
- Exactness vs the original PyTorch pipeline: 4 clips (speech of three lengths + 3 s of noise, 2696 frames): max |ONNX − PyTorch| salience 5.66e-06, max pitch difference 0.00020 cents (1 cent = 1/100 semitone), voicing agreement 100.0 %.
- Voicing threshold: 0.35 (chosen on PTDB-TUG's dev speakers, lowest F0 frame error; the upstream default is 0.03).
Accuracy on PTDB-TUG's 16 test speakers (400 utterances per condition), in %:
| condition | RPA ↑ | RCA ↑ | GPE ↓ | VDE ↓ | FFE ↓ | VR ↑ | VFA ↓ |
|---|---|---|---|---|---|---|---|
| clean | 91.15 | 92.09 | 1.36 | 2.78 | 3.09 | 94.34 | 1.89 |
| white noise 10 dB | 90.98 | 92.04 | 1.51 | 2.84 | 3.18 | 94.50 | 2.01 |
| white noise 0 dB | 89.04 | 90.26 | 1.76 | 3.17 | 3.56 | 93.14 | 2.01 |
| pink noise 10 dB | 89.41 | 90.59 | 1.70 | 3.10 | 3.48 | 93.10 | 1.91 |
| pink noise 0 dB | 84.31 | 85.57 | 1.94 | 4.05 | 4.47 | 88.98 | 1.87 |
CPU speed (onnxruntime, x86 server CPU; RTF = compute seconds per second of audio; peak RSS is the whole Python process, librosa and numpy included):
| threads | audio | RTF (compute s / audio s) | peak RSS |
|---|---|---|---|
| 1 | 10 s | 0.0559 | 1079 MB |
| 4 | 10 s | 0.0166 | 1052 MB |
| 1 | 30 s | 0.0539 | 1089 MB |
Comparison on PTDB-TUG (F0 frame error, %, lower is better)
FFE = (voicing-decision errors + gross pitch errors > 20 %) / all frames — the single number that counts both "is this frame voice" and "how wrong is the pitch". Reference: PTDB-TUG's laryngograph-derived f0 (RAPT, 10 ms). Speakers F01 F02 M01 M02 were used ONLY to pick each method's voicing threshold and frame lag (clean audio, lowest FFE); the table is the other 16 speakers, 400 utterances per condition, scored once. Noise is added at the stated SNR over the whole utterance. Best per row in bold.
| condition | RMVPE | FCPE | CREPE full | CREPE tiny | PESTO | pyin (librosa 0.11) |
|---|---|---|---|---|---|---|
| clean | 3.09 | 2.77 | 6.43 | 6.33 | 5.58 | 7.72 |
| white noise 10 dB | 3.18 | 2.95 | 5.98 | 6.07 | 5.99 | 8.32 |
| white noise 0 dB | 3.56 | 3.39 | 6.92 | 7.21 | 7.46 | 16.92 |
| pink noise 10 dB | 3.48 | 3.30 | 6.59 | 6.41 | 6.54 | 8.36 |
| pink noise 0 dB | 4.47 | 4.78 | 9.48 | 9.31 | 8.99 | 17.18 |
Metrics per condition for every method, the chosen thresholds/lags and the CPU timings are in
benchmark.json; the harness is export/evaluate.py.
How it was exported
export/export.py (torch 2.8, legacy ONNX exporter, opset 17). The STFT is two conv1ds with
fixed DFT kernels (torch.stft's complex output doesn't export), equal to torch.stft to float32
rounding. export/decode.py is the decoder the parity check uses. RMVPE ships fp16 weights: the graph stores every fp16-exact weight as fp16 with a Cast to fp32 in front, so it computes in fp32 with bit-identical results at half the file size.
License and credits
Weights: RVC-Project rmvpe.pt (MIT, Copyright (c) 2023 liujing04, 源文雨, Ftps), from the original Dream-High/RMVPE (Apache-2.0, included as LICENSE-RMVPE-original). Network code rmvpe_net.py is RVC's, verbatim.
ONNX conversion, evaluation and packaging by Scicom.