A model that separates a song into the instrumental, the lead vocals and the
backing vocals in one pass, made for Musetric.
It is distilled anew from Aname's
Mel-Band RoFormer Duality
for the vocals, with a second mask that splits its vocals into lead and
backing, distilled from UVR MDX-Net KARA 2 of
Ultimate Vocal Remover.
It is narrower than the previous revision of this repository,
50840af,
and scores as well on true stems: 116.0 M parameters against 135.0 M, and
its core peaks at 1232 MiB of GPU memory against 1492 MiB in a browser, at
the same speed.
These are the PyTorch weights in fp32 and their config: the original model,
from which the app's ONNX core is exported. The class that reads them is
LeadBackingRoformer
of musetric/lab.
Intended:
Limitations:
import yaml
from safetensors.torch import load_file
from musetric_toolkit.separate_audio.roformer.lead_backing_roformer import LeadBackingRoformer
config = yaml.safe_load(open("config.yaml"))
model = LeadBackingRoformer(**config["model"]).eval()
model.load_state_dict(load_file("model.safetensors"))
stems = model.separate(audio) # audio: float32 [batch, 2, samples] at 44.1 kHz, an 11 s window
# stems["vocals"], stems["backing"], stems["lead"] = vocals - backing, stems["instrumental"] = audio - vocals
model.net_forward(stft_repr) is the neural network core alone: a real STFT
[batch, 2050, frames, 2] in, two masks per band out, the vocals mask over the
mix and the backing mask over the vocals; stem_spectra averages them per bin
and applies them.
| File | Size | SHA256 |
|---|---|---|
model.safetensors |
464,168,152 B | 1f2dfa76eb7968bbaa004850b0f4654691da21cf60b4e5df3f9a2784ab765845 |
config.yaml |
1,170 B | 9ffaeac11097e3a4271bd04b557684d54fe77c87838908d0797bbdce8b512c1f |
training-data.csv |
2,072,765 B | baab7435e3c168ee87426ac443ff9c7ce8443e443aa0406e7a2c987f6dac11e4 |
116.0 M parameters: 90.5 M for the vocals and a split mask estimator of 25.5 M. Against Duality:
training-data.csv.Against the previous revision, 50840af. SDR over whole tracks; paired differences are the mean of this model
minus the previous one per track, with a 95 % bootstrap interval.
Lead and backing against true stems. The test set is 41 ccMixter a
cappellas whose lead and backing are separate uploads, by 11 uploaders under
CC BY and CC0, 18 of them also over the upload's own instrument stems
(data-ccmixter-voices of musetric/lab). The backing is scored where it is no
more than 20 dB below the vocals.
| Set | Stem | This model, median | Previous, median | Paired difference |
|---|---|---|---|---|
| Mixes (18) | Lead | 6.20 dB | 5.68 dB | +0.18 dB [-0.02, +0.45] |
| Mixes (18) | Backing | 1.90 dB | 1.35 dB | +0.25 dB [-0.00, +0.57] |
| Mixes (18) | Vocals | 12.76 dB | 13.21 dB | +0.13 dB [-0.06, +0.28] |
| A cappellas (41) | Lead | 3.65 dB | 3.67 dB | +0.04 dB [-0.04, +0.15] |
| A cappellas (41) | Backing | 0.49 dB | 0.51 dB | +0.02 dB [-0.09, +0.14] |
On the tracks where the backing of the student followed by KARA 2 scores above 2 dB, where KARA 2 really splits, this model's backing is 0.37 dB [+0.03, +0.90] above the previous model's on the 8 mixes and 0.06 dB [-0.40, +0.26] below on the 10 a cappellas.
Vocals on MUSDB18-HQ, test, 50 tracks, 11 s windows every 8 s: median SDR 11.24 dB against 11.23 dB; paired, -0.03 dB [-0.15, +0.08].
Memory and speed. The ONNX core in Chrome on WebGPU on a desktop GPU (RTX 5070 Ti), one 11 s window: the peak of its GPU buffers 1232 MiB against 1492 MiB, 149 ms for both (median of ten runs). The core's weights file is 235 MB against 273 MB. On onnxruntime CUDA the core takes 68.5 ms against 59.3 ms; a whole song in PyTorch fp16, Rxbyn – Bad Side (3:37, CC BY), 2.49 s against 2.45 s.
The work is described in musetric/musetric#1088.
Aname-Tommy/Mel-Band-Roformer_Duality
@ 07a189ffdceb69db0ddacc5175569bef59850016, duality_v1.ckpt,
Apache-2.0. It is a fine-tune of
SYH99999/MelBandRoformerMergedSYHFTBeta1
(MIT), which builds on KimberleyJSN/melbandroformer
(MIT).UVR_MDXNET_KARA_2.onnx of the
Ultimate Vocal Remover model release,
trained by the developers of Ultimate Vocal Remover,
MIT; the app's export of it is musetric/uvr-mdxnet-kara2-onnx.lucidrains/BS-RoFormer, MIT.release-voices and the lead/backing recipe of musetric/lab at
6daf2f3.Apache-2.0, as Duality's weights it starts from. The split is distilled from KARA 2, MIT, with credit to Ultimate Vocal Remover and its developers. We do not claim authorship of the teachers; the compression, the split and the training are ours.
@article{wang2023melbandroformer,
title={Mel-Band RoFormer for Music Source Separation},
author={Wang, Ju-Chiang and Lu, Wei-Tsung and Won, Minz},
journal={arXiv preprint arXiv:2310.01809},
year={2023}
}