MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces

MSVAE is a music variational autoencoder with a structured, continuous latent space. This repository contains the Stage II reconstruction checkpoint: it accepts stereo audio at 24 kHz and produces stereo audio at 48 kHz. The encoder runs at 24 kHz, while the decoder generates the 48 kHz waveform. This is an audio reconstruction model, not a standalone text-to-music generator.

The model combines a frozen music self-supervised encoder with an acoustic encoder through a gated residual posterior. During training, separate parts of the 64-dimensional latent are aligned with compositional and linguistic-vocal references. The released checkpoint is for inference; training-time alignment features are not required to reconstruct audio.

Files

The Stage I 24 kHz-to-24 kHz checkpoint is not included here. The model.pt file is approximately 721 MiB.

Requirements

Inference requires the accompanying MSVAE code and its Python environment. The checkpoint is not directly loadable through transformers.AutoModel. The model also uses OpenMuQ/MuQ-large-msd-iter, which the inference code loads separately with MuQ.from_pretrained(...); MuQ weights are not redistributed in this repository. Please follow the MuQ model license for that dependency.

The code repository, installation instructions, paper, and audio demo links will be added when their public releases are available.

Reconstruction

From the root of the accompanying MSVAE code repository, after installing its runtime dependencies:

CKPT_PATH=/absolute/path/to/model.pt \
bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0

The third positional argument selects GPU 0. The script resamples input audio to 24 kHz before encoding and writes a 48 kHz stereo WAV file under ./reconstructions. Mono input is duplicated to stereo. By default, it samples the VAE latent with seed 42. For deterministic decoding of the posterior mean, append --latent-mode mean:

CKPT_PATH=/absolute/path/to/model.pt \
bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0 --latent-mode mean

The default inference configuration in the code repository, src/config/svae/example_new.yml, has the same model settings as config.yaml here. Its DAC.bigvgan_conf path is resolved relative to the code repository root, where the same decoder JSON is provided.

Intended use and limitations

This checkpoint is intended for music reconstruction research and listening comparisons. The 48 kHz output is generated from a 24 kHz input; high-frequency detail is inferred by the decoder rather than recovered from the original waveform. Performance on other audio domains has not been established here. Use only input audio you are entitled to process and publish.

License and citation

The MSVAE checkpoint license and paper citation have not yet been specified. Add the chosen license and public paper reference before publishing this model repository. The external MuQ weights have their own CC BY-NC 4.0 license.