w2v-bert-2.0-music-lyrics-ctc

A character-level CTC lyrics model for full music mixes, i.e. vocals with the band, no source separation. It's fine-tuned from w2v-bert-2.0-music, with the encoder unfrozen. It is multilingual: the 921-token vocabulary covers Latin, Cyrillic, Greek, Arabic, Hebrew, Thai, kana/kanji and Hangul (decomposed into jamo).

Uses:

• Lyrics transcription (greedy CTC decoding).
• Forced alignment of known lyrics to audio (CTC Viterbi), which gives word and syllable onsets.
• Frame-level vocal activity and vocal F0/voicing from the auxiliary heads.

import torch
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor

processor = Wav2Vec2BertProcessor.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc")
model = Wav2Vec2BertForCTC.from_pretrained("mrfakename/w2v-bert-2.0-music-lyrics-ctc").eval()

# 16 kHz mono; up to ~30 s per call (see "Long audio" below)
inputs = processor(audio_16khz_mono, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits              # (1, frames at 50 Hz, 921)
text = processor.batch_decode(logits.argmax(-1))[0]

Output format

• Lowercase, NFKC-normalised text with punctuation removed. Apostrophes are kept inside words.
• Hangul comes out as conjoining jamo. Use unicodedata.normalize("NFC", text) to recompose syllables.
• Token 0 is the CTC blank, and | is the word delimiter / filler.
• The filler also absorbs vocals that aren't lyrics (ad-libs, humming). For alignment, put | between words and at both ends of the target sequence.
• Section and performance tags ([Chorus], (backing vocals)) are not transcribed.

Long audio

The model was trained on windows of up to 30 s. For longer audio, run overlapping windows (e.g. 30 s with 2 s of context on each side), keep the centre of each window, and concatenate the logits before decoding or aligning.

Auxiliary heads

aux_heads.safetensors holds two linear heads that operate on the final hidden states (output_hidden_states=True, last layer):

• vad.{weight,bias}: 1 logit per frame, vocal activity.
• f0.{weight,bias}: 361 outputs per frame. Index 0 is the voicing logit; indices 1..360 are vocal pitch bins of 20 cents each (bin b → 10 * 2 ** ((20 * b + 1997.38) / 1200) Hz, ≈32 Hz–2 kHz).

Training

Fine-tuned from w2v-bert-2.0-music with weak supervision from whole-song lyrics:

1. Full-song CTC bootstrap.
2. Forced alignment and line-based chunking; the worst-aligned 15% of chunks are dropped.
3. CTC on ≤ 30 s chunks, plus random windows cut anywhere in a song, so the model doesn't rely on inputs starting at a line boundary.

The vocal-activity and F0/voicing heads were trained jointly as auxiliary losses (weight 0.1 each).

Evaluation

Held-out songs, greedy decoding, no language model:

The reference lyrics are imperfect: the model often recovers words the references miss, so the true error is somewhat lower.

Forced-alignment timing: a word/syllable onset evaluation will be added here.

• CTC emissions are peaky, so onsets from forced alignment are usable, but word offsets (ends) are not reliable.
• For boundaries within about 20 ms, a dedicated boundary model is needed on top of this encoder.