SingMelody — production note-transcription weights

Weights for SingMelody, which transcribes notes for singing voice: given audio and a finished alignment, it gives each word one or more pitches — onsets, offsets and MIDI — without moving a single phone boundary.

This repository holds one model.

Quick start

You do not download these files by hand. SingMelody resolves a model name to a local path, fetching and checksumming it on first use:

git clone https://github.com/pymaster17/SingMelody
cd SingMelody
uv sync && source .venv/bin/activate

python transcribe_one.py --backend game song.wav song.TextGrid --out notes/

The first run downloads this bundle into ~/.cache/singmelody/checkpoints (override with $MELODY_CACHE_DIR) and verifies every file against the SHA256 pinned in the model spec. From Python:

from singmelody.api import GAMETranscriber, transcribe_results

results = transcribe_results(alignments, GAMETranscriber())          # default
results = transcribe_results(alignments, GAMETranscriber(language="ja"))

See docs/getting_started.md in the repository for the full setup.

game-multilingual-singlestep-v1

A bundle of three files that must stay together — the loader reads the weights' two siblings:

model-latest-steps=0030000-epochs=0060.ckpt   the weights
config.yaml                                   the resolved model config
lang_map.json                                 language -> embedding id

• Backbone: openvpi GAME's JEBF encoder, segmenter and estimator, vendored unchanged.
• Training: M4Singer + all nine GTSinger languages, ~110 h, 30000 steps / 60 epochs, 4×A6000 DDP. Word-tier masking with dropout p=0.5, so one model serves both regimes: completing a given word tier, and predicting notes with no word tier at all.
• Inference is one forward pass at a constant t. This is not a D3PM model — the corruption process is replaced by the fixed word-tier mask that transcription actually presents. There is no sampling loop and no RNG, so reproducibility is a property of the construction rather than of a pinned seed.
• Languages: zh en de fr es it ja ko ru. The language= you pass must be in that table; an unknown code raises rather than being guessed.

What it is good at, and where it is not

Conditional regime — you have a word tier, which is the production path. Usable in all nine languages. melisma F1 by language, on a 5030-clip held-out set:

zh 0.7999 · m4singer 0.7713 · fr 0.7439 · en 0.7353 · it 0.7181
ru 0.6212 · de 0.6174 · ko 0.6118 · es 0.5813 · ja 0.5685

That column cannot be read across rows. It scores one class — word-internal note onsets — whose annotation density varies 5.2x between languages (ko/ja 0.082, it 0.425), and GTSinger's "word" is an orthographic character in zh/ja/ko but a lexical word in the six European languages. The two families are not doing the same task. Pitch is never the bottleneck: raw_pitch_accuracy is 0.79–0.89 across the whole table.

Free regime — no word tier, the model predicts the note layer itself. Solid in Chinese (QER 0.2318 / 0.2453), and weak outside it: QER 0.63–0.80 across the other eight languages. That is a cliff, not a gradient, and roughly half of it is that the task is underdetermined in polysyllabic languages — the missing information is lexical and is not among the model's inputs. Use the conditional regime outside Chinese.

Chinese is not traded away for the other eight

Measured against a Mandarin-only arm on a held-out set pinned clip-for-clip to the same 323 Chinese clips, so the comparison is strictly single-variable:

Flat on every metric in both regimes. That is why this — rather than the Mandarin-only arm — is the repository's default GAME checkpoint: the same weights already annotated 230,936 songs of production corpus, so the default path and the production path are one model.

Notes on the file

It is a Lightning training checkpoint, optimizer state included, uploaded byte-identical to what training wrote. Nothing was stripped, so a downloaded copy and a locally trained one hash the same and the SHA256 in the model spec is the one the repository's own checkpoint inventory records. model-latest rather than a validation-selected best is deliberate: this run reaches its plateau by step 6000 and has no trend over the remaining 24000 steps, so the final state is the one to use.

Not published here

• game-mandarin-singlestep-v1 — the Chinese-only arm. Its spec ships with the repository and its numbers are in docs/experiments/, but the weights are not released: this model matches it on Chinese, so there is nothing it buys. Retrain it with configs/game/train_pool_dropout.yaml if you need the exact arm a historical table was measured on.
• rosvot-pool-v1 / rwbd-pool-v1 — SingMelody's second backend, ROSVOT retrained on the same pool. It measures within 0.5–1 point of GAME, which is why the repository keeps both and defaults to neither at the backend level. The weights are not released; bench/slurm/train_rosvot_pool.sh is the recipe.

Licence

MIT, matching the repository and both upstream projects. Vendored third-party code and its licences are recorded in THIRD_PARTY_NOTICES.md and LICENSES/.