Vocos (mel, 24 kHz), ONNX (float16 weights)

A sound made again from its mel spectrogram: charactr's vocos-mel-24khz, Vocos (Siuzdak, "Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis", ICLR 2024, arXiv:2306.00814), trained on LibriTTS: a ConvNeXt backbone (8 layers, dim 512) predicting each STFT frame's magnitude and phase. The network between its features and its iSTFT, as @audio/neural-vocode runs it and audio's vocos() uses it.

Float16 weights, float32 compute (onnxruntime casts them at load); the float32 export it is made from is 54.2 MB.

Source

• Checkpoint: pytorch_model.bin (SHA-256 97ec976ad1fd67a33ab2682d29c0ac7df85234fae875aefcc5fb215681a91b2a) and its config.yaml, charactr/vocos-mel-24khz at 0feb3fdd929bcd6649e0e7c5a688cf7dd012ef21 (MIT).
• Model code: gemelo-ai/vocos (MIT).
• Export: scripts/export.py --verify in @audio/neural-vocode, then compact.py --model vocos --as fp16 from @audio/neural-separate.

Licence and attribution

MIT (LICENSE), as the checkpoint's repository states. The model and its weights by Hubert Siuzdak (Charactr Inc.); ONNX export and float16 weights by audiojs.

Graph

The graph is what Vocos.decode computes between its features and its iSTFT: the backbone and the head's linear layer. The features (torchaudio's MelSpectrogram) and the iSTFT (torch.istft, Hann of 1024, centered) are computed outside it, in @audio/neural-vocode. --verify: the graph within 6e-5 of the PyTorch forward on noise and tones.

import vocode from '@audio/neural-vocode'
let y = await vocode(samples, { sampleRate: 44100 })   // as long, at 44.1 kHz; nothing above 12 kHz

Quality

Through @audio/neural-vocode, on a 2 s fixture: 0.050 dB of log-mel distance from PyTorch's Vocos.decode (0.049 for the float32 graph), where PyTorch is 0.047 dB from itself between float32 and float64 features and Vocos changes the sound by 3.7 dB. On a 36 s TTS take at 44.1 kHz, its cepstrum's peaks at 20 and 30 ms (a vocoder's frames ringing on), 0.020 and 0.010, fall to 0.0003 and 0.0005, the background's level. A speech vocoder: music comes back altered.

@inproceedings{siuzdak2024vocos,
  title     = {Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis},
  author    = {Siuzdak, Hubert},
  booktitle = {ICLR},
  year      = {2024},
  eprint    = {2306.00814},
  archivePrefix = {arXiv}
}