ONNX export of musetric/mel-band-roformer-duality-kara2-student
@ 8f928dd73899dd9279a878d732841919a98e61e6, the model that separates a song into the instrumental, the lead
vocals and the backing vocals in one pass, packaged for the
musetric packages/ai runtime
(onnxruntime-web on WebGPU). Apache-2.0, as the weights it is exported
from.
The graph is the neural network core, with the input of the vocals
student's core musetric/mel-band-roformer-duality-student-onnx
and two masks out instead of one: it takes a precomputed STFT representation
and returns per-bin complex masks, the vocals mask over the mix and the backing
mask over the vocals. The mel-band gather/average tables are baked into the
graph; STFT, iSTFT, chunking and complex packing run host-side.
Intended:
onnxruntime-web.Out of scope:
masks, and runs iSTFT (see musetric packages/ai).Limitations:
The session runs the core; the host supplies stft_repr and consumes masks.
import * as ort from 'onnxruntime-web/webgpu';
// .onnx and .onnx.data must sit in the same directory; .data loads automatically.
const session = await ort.InferenceSession.create('duality_kara2_student_core_t1100.onnx', {
executionProviders: [{ name: 'webgpu', storageBufferCacheMode: 'simple' }],
});
// stftRepr: Float32Array of shape [1, 2050, 1100, 2], produced host-side from one
// ~11 s audio chunk (n_fft=2048, hop=441, 44.1 kHz, stereo).
const input = new ort.Tensor('float32', stftRepr, [1, 2050, 1100, 2]);
const { masks } = await session.run({ stft_repr: input });
// masks: float32 [2, 2050, 1100, 2]. vocals = stft * masks[0]; backing = vocals * masks[1]
// (complex products); then iSTFT of each; lead = vocals - backing, instrumental = mix - vocals.
One graph. Keep the .onnx next to its .onnx.data.
| File | Size | SHA256 |
|---|---|---|
duality_kara2_student_core_t1100.onnx |
6,230,015 B | 86a884e0225233a7d77f88e60d9a7e2febdb0d386d19ddc9476fcf51ed858228 |
duality_kara2_student_core_t1100.onnx.data |
235,400,588 B | 176673d2446d8578976dd1a17ebca5fa0651dd23446967ff44fad5bf22c45d5c |
Signature — opset ai.onnx 23, IR 10, no com.microsoft import. fp16
weights, fp32 graph I/O, T = 1100:
| Tensor | Type | Shape | Meaning |
|---|---|---|---|
stft_repr (in) |
float32 | [1, 2050, T, 2] |
batch, freq*2, time, complex |
masks (out) |
float32 | [2, 2050, T, 2] |
per-bin complex masks: 0 the vocals over the mix, 1 the backing over the vocals |
The vocals path is written as the core of
musetric/mel-band-roformer-duality-student-onnx,
see its card, with this model's widths: a 288-wide stream, feed-forwards of
768 in pairs 0-1 and 1536 in pairs 2-5, and pairs 2-5 on frames averaged by
2. The split estimator runs per band on the same pooled output of the
transformer stack: a 288 x 768 linear layer, a tanh and a linear layer to
twice the band's size, gated, in fp16 like the vocals estimator. Its masks go
through the same mel-band average as the vocals mask and leave the graph as
its second output.
Against the PyTorch model in fp32 on CUDA at the same window, the core on the CUDA execution provider (TF32 off), its masks applied to the window's STFT and inverted to vocals, backing and lead. Each song is peak-normalised to 0.9 and scored on its loudest 1100-frame window. The songs are every CC BY and CC BY-SA track of JamendoLyrics.
| Song | Genre | License | Window | Vocals | Backing | Lead |
|---|---|---|---|---|---|---|
| Lower Loveday – Is It Right? | Indie | CC BY-SA | 156.7 s | 63.7 dB | 58.0 dB | 62.8 dB |
| Rxbyn – Bad Side | RNB | CC BY | 29.5 s | 64.2 dB | 49.9 dB | 64.2 dB |
| Cortez – Feel (Stripped) | Pop | CC BY | 195.8 s | 66.5 dB | 55.6 dB | 65.8 dB |
| I'M FRESH! YOU'RE PRETTY! – CHRISTMAS AVEC TOI | Pop | CC BY | 180.4 s | 68.1 dB | 67.2 dB | 45.3 dB |
| NANDOMALO – Esencia | Folk | CC BY-SA | 63.3 s | 70.2 dB | 55.8 dB | 61.6 dB |
| LOS ROMBOS – Fantasma | Folk | CC BY-SA | 67.1 s | 66.5 dB | 49.7 dB | 66.4 dB |
| Durch Dick und Duenn – Freifliegen | Rock | CC BY-SA | 140.8 s | 64.0 dB | 59.7 dB | 64.0 dB |
| Jonny M – Keine Lust | Reggae | CC BY-SA | 189.4 s | 65.0 dB | 50.9 dB | 63.3 dB |
| FLO – l'abandon | Country | CC BY | 163.7 s | 67.0 dB | 61.0 dB | 66.9 dB |
| KPTN – Le musee d'air contemporain | Jazz | CC BY | 80.0 s | 68.5 dB | 54.7 dB | 68.5 dB |
| LAW' – Les files d'attente | Pop | CC BY-SA | 38.6 s | 68.2 dB | 61.9 dB | 68.5 dB |
| LAW' – Mere nature | Rock | CC BY-SA | 140.2 s | 67.5 dB | 50.9 dB | 67.5 dB |
| WILSON WAY – Te Recuerdo | Hip-Hop | CC BY-SA | 83.1 s | 65.5 dB | 54.7 dB | 64.7 dB |
SNR, worst / median / best: vocals 63.7 / 66.5 / 70.2 dB, backing 49.7 / 55.6 / 67.2 dB, lead 45.3 / 64.7 / 68.5 dB; no NaN. The window column is where the scored window starts in the song.
In Chrome on WebGPU on a desktop GPU (RTX 5070 Ti), one 11 s window: the peak
of the core's GPU buffers 1232 MiB against 1492 MiB for the core of the
previous revision, 0555bb1, 149 ms for both
(median of ten runs). On onnxruntime CUDA 68.5 ms against 59.3 ms. The
weights file is 235 MB against 273 MB.
musetric/mel-band-roformer-duality-kara2-student
@ 8f928dd73899dd9279a878d732841919a98e61e6 (model.safetensors, config.yaml), Apache-2.0; its lineage,
training and evaluation are on its card.LeadBackingRoformer of musetric/lab.scripts/onnx/roformer/build_lead_backing_onnx.py in
musetric/lab at 6daf2f3, with
--core-only --fuse-rmsnorm --attn-block 128 --all-fp16 --frames 1100 --skip-gate --split-rows 2 --stream-slabs 4;
then split_concat_webgpu.py and fuse_webgpu_kernels.py; it passes
fp16_epsilon_audit.py and dispatch_rows_audit.py. The recipe is
converting.md there.Apache-2.0, as the weights it is exported from; the split is distilled from UVR MDX-Net KARA 2, MIT, with credit to Ultimate Vocal Remover and its developers.
@article{wang2023melbandroformer,
title={Mel-Band RoFormer for Music Source Separation},
author={Wang, Ju-Chiang and Lu, Wei-Tsung and Won, Minz},
journal={arXiv preprint arXiv:2310.01809},
year={2023}
}