MOSS Voice-Acting — Sports-Commentator LoRAs

⚠️ Do not merge this adapter into the base weights

merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint" script will destroy the model irrecoverably. This is not a performance caveat. Read this before you write a deployment script.

Why

This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23 target modules. In this architecture those output heads are weight-tied to the input embeddings: tie_weights() sets

audio_lm_heads[i].weight  IS  audio_embeddings[i].weight     # the same tensor, not a copy
text_lm_head.weight       IS  transformer.embed_tokens.weight

They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head weight, it writes that delta straight into the embedding table at the same time. The model then reads its own inputs through a matrix that has been shifted by an output-side correction. Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the original values are gone.

Verify it yourself in three lines

Do not take our word for it:

m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True  -> same storage, merging corrupts the embeddings

What to do instead

Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:

from peft import PeftModel

model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()

def set_weight(model, name, w):
    """Scale one named adapter's contribution.  alpha/r is its own base scaling."""
    for module in model.modules():
        scaling = getattr(module, "scaling", None)
        if isinstance(scaling, dict) and name in scaling:
            if not hasattr(module, "_base_scaling"):
                module._base_scaling = {}
            module._base_scaling.setdefault(name, scaling[name])
            scaling[name] = module._base_scaling[name] * float(w)

set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"])      # several adapters can be active at once

This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r), which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same arithmetic, in a different order. You give up a small amount of inference speed and you keep the ability to change the weight, stack several adapters, or turn one off. Nothing about the sound changes.

If you are stacking adapters

Set each one's scaling separately and activate them together with model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add adapters deliberately and measure.

If you maintain code that merges

A regex over module names is not enough — the reliable test is identity of storage. Group the modules by weight.data_ptr() and refuse to merge into any group with more than one member. lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.

---

PEFT/LoRA adapters that push laion/moss-tts-local-transformer-4.55b-voice-acting-v2 toward energetic live sports commentary — the shouted, fast, rising-intensity register of a broadcaster calling a goal, a world record or a knockout as it happens.

Two training runs are published here, both swept over ranks 16/32/64 × epochs 1/2/3/8 and evaluated on the same 16 held-out English prompts with a real-audio ceiling control.

👉 Use the real_* adapters

real_r64_e8 is the default. It and its siblings real_r32_e2 / real_r32_e1 were trained on the 468 real German Mediathek broadcast segments only — the 820 synthetic English generations were dropped. They beat the mixed-data adapters in 11 of 12 matched configurations, tie in 1, lose in none (paired Wilcoxon p = 0.0010), and 7 of 12 reach a perfect 2.000/2.

Why real_r64_e8 and not the lowest-WER cell: it was picked by listening. It is the most emotionally convincing of the set — the one that actually sounds like an energetic human broadcaster rather than a model performing one. The automatic metrics agree on the direction (arousal 5.00, ranting 5.31, commentary score 0.943 — the highest of every cell in both runs) but they ranked real_r32_e2 first on WER, and WER is not what makes commentary good. On this stack automatic metrics have now failed to rank checkpoints three separate times.

The mixed-data adapters (r64_e1, r64_e8, r32_e1) are kept for reproducibility and because the comparison between them is the interesting part — see Real data only beats the mix.

🎧 Listen to the full evaluation — all 13 cells, base, and real human commentary

---

Real data only beats the mix

Same script, same hyperparameters, same seed, same held-out prompts. The only change: drop the synthetic English half of the training data. Note that this also makes the adapter German-only (every Mediathek row is labelled language="German") while the evaluation prompts are English — so this measures whether real commentary transfers the register across a language boundary.

It does, and better than adding synthetic English data did.

The mechanism is visible in the acoustics: real-only training pushes exactly the commentator dimensions — arousal, ranting, emphasis — much harder, and it does not cost intelligibility (WER is statistically unchanged, so this is not a "German accent fools the judge" artifact). What it costs is genuineness and blend: the adapter is louder and more performed.

The base cell is identical in both runs (1.750), which is the check that the two evaluations are comparable at all — generation is seeded, so the shared anchor should not move, and it did not.

The honest limit on this comparison

The real-only cells hit sd = 0.000 with 100 % of clips at the top score. The 0–2 scale is completely exhausted. Worse, the arithmetic says the design could never have produced a Bonferroni-significant single cell: a perfect cell against this base sample gives a permutation p = 0.00515, and the threshold for 12 comparisons is p < 0.00417. No achievable result could have cleared it. The paired across-cell test (p = 0.0010) is the one that carries weight, because it compares the two runs to each other rather than each cell to base.

---

⚠️ Read this before you use them: the base model is already good at this

We measured the honest thing, so here it is up front.

n = 32 clips per cell, 456 clips total, judged by gemini-3-flash on the same 0/1/2 rubric used to filter the training data. 0 unparsed.

Three things follow, and none of them should be glossed over:

1. The base model (1.750) is statistically indistinguishable from real human sports commentary (1.775). The metric's ceiling is ≈1.8 and the un-adapted model is already there. There is very little headroom for an adapter to occupy.
2. No cell survives multiple-comparison correction. Two cells clear p < 0.05 raw, but with 12 comparisons against base the Bonferroni threshold is p < 0.0042. Treat the ranking above as suggestive, not established.
3. A 0–2 absolute scale was the wrong instrument. Every cell lands between 1.75 and 1.97 and 75–97 % of clips get the top score — the scale is compressed against its ceiling. A follow-up should use pairwise A/B preference against base, which stays sensitive when everything already sounds like sports commentary.

Practical guidance: try the base model first. Reach for these adapters if you want the register pushed harder and more consistently (the % rated 2 column is where the difference is most visible: 97 % vs 75 %), not because the base model fails at the task.

Validation loss did not predict the listening result

The training log says epoch 1 is best and epoch 8 badly overfits — at rank 64, val loss goes 4.4237 → 5.4369, a 0.905 regression. Listeners rate r64_e1 and r64_e8 identically at 1.969. That is why both are published here: whatever epoch 8 lost in validation loss, it did not lose in how the audio sounds.

This is the third time on this model stack that val loss failed to rank checkpoints the way a listener does. Rank on generation-side metrics, not on loss.

---

Which adapter to pick

Take real_r64_e8 — it is the default for energetic, real-sounding sports commentary. Drop to real_r32_e2 if you need the lowest WER or half the download.

---

Where everything lives

---

Quickstart

import torch, soundfile as sf
from transformers import AutoProcessor, AutoModel
from peft import PeftModel

BASE    = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
CODEC   = "OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"
ADAPTER = "real_r64_e8"               # default; see "Which adapter to pick"

# AutoModel, NOT AutoModelForCausalLM -- MossTTSLocalConfig is not registered for the
# CausalLM auto-class and from_pretrained raises "Unrecognized configuration class".
proc  = AutoProcessor.from_pretrained(BASE, trust_remote_code=True, codec_path=CODEC)
model = AutoModel.from_pretrained(
    BASE, trust_remote_code=True, dtype=torch.bfloat16,
    attn_implementation="sdpa",       # flash-attn 2.x is incompatible with this model
).cuda().eval()

pm = PeftModel.from_pretrained(
    model, "laion/moss-sports-commentator-lora",
    subfolder=ADAPTER, adapter_name=ADAPTER,
).eval()

# `instruction` is the whole director's note; `text` is ONLY the spoken words.
# Empty fields render as the literal string "None", so fill them deliberately.
GENERAL = ("A voice that is extremely energetic and aroused; extremely fast and rapid in tempo; "
           "extremely emphatic and projected; strongly ranting and worked up; adult; strongly "
           "masculine; wide-ranging in pitch; genuine and spontaneous rather than performed; "
           "volatile and unstable.")
CUE = ("Extremely energetic and aroused, extremely fast and rapid in tempo, adult, strongly "
       "masculine, extremely emphatic and projected, genuine and spontaneous")

sents = ["He's through, he's one on one — and he's buried it!",
         "In the last minute of the final!",
         "This entire stadium has lost its mind!"]
script = " ".join(f'({CUE}) "{s}"' if i != 1 else f'({CUE}) [pause 0.3s] "{s}"'
                  for i, s in enumerate(sents))
instruction = f"GENERAL: {GENERAL}\nSCRIPT:\n{script}"
text = " ".join(sents)

# The length model that was fitted for English on this stack: words dominate ~4:1,
# ~2.65 words/s, ~12.5 tokens/s.
tokens = int(round(len(text.split()) / 2.65 * 12.5))

conv  = [[proc.build_user_message(text=text, instruction=instruction,
                                  language="English", tokens=tokens)]]
batch = proc(conv, mode="generation")

with torch.no_grad():
    out = pm.generate(
        input_ids=batch["input_ids"].cuda(),
        attention_mask=batch["attention_mask"].cuda(),
        max_new_frames=300, do_sample=True,
        text_temperature=0.7, text_top_k=50, text_top_p=1.0,
        audio_temperature=1.0, audio_top_k=30, audio_top_p=0.95,
        audio_repetition_penalty=1.1,
    )

msg = proc.decode(out)[0]
w = msg.audio_codes_list[0].cpu().float().numpy()   # ALREADY a waveform -- see Traps
if w.ndim > 1:
    w = w.mean(0)
sf.write("commentary.wav", w, 48000)

eval_prompts.json in this repo holds all 16 held-out evaluation prompts in exactly this shape (8 scenes × male/female voice, 16 sports), so you can reproduce the table above.

Adapter strength (merge scale)

The delta is added scaled by alpha / r — 2.0 for every adapter here. Multiply by a dose λ:

from peft.tuners.lora import LoraLayer

# Capture the untouched scaling ONCE. Reading the current value and multiplying makes the
# scale compound on every change and silently drift.
base_scaling = {n: dict(m.scaling) for n, m in pm.named_modules() if isinstance(m, LoraLayer)}

def set_dose(adapter: str, lam: float):
    for n, m in pm.named_modules():
        if isinstance(m, LoraLayer) and adapter in m.scaling:
            m.scaling[adapter] = base_scaling[n][adapter] * lam

set_dose(ADAPTER, 0.5)

The evaluation above was run at λ = 1.0. On a related sweep, ECAPA speaker similarity to a reference clip fell 0.62 → 0.57 → 0.50 → −0.03 at λ = 0 / 0.5 / 1.0 / 1.5, against a 0.105 unrelated-speaker floor — so if you are voice-cloning from a reference, keep λ ≤ 0.5. For a generic commentator voice with no reference to preserve, λ = 1.0 is what was measured.

Stacking

Swapping the active adapter costs ~0.021 s across 268 modules, so combining is cheap:

pm.load_adapter("TTS-AGI/moss-emotion-loras-v3", subfolder="Anger", adapter_name="Anger")
pm.base_model.set_adapter([ADAPTER, "Anger"])
set_dose(ADAPTER, 0.8); set_dose("Anger", 0.3)

Untested combination — the evaluation here covers the sports adapter alone.

Traps

• audio_codes_list on the output side already holds a decoded waveform, not codes. Decoding it again yields exactly 0.16 s of pad per clip — a silent failure that still writes plausible-looking WAV files.
• Pass the sampling parameters explicitly. generate()'s continuation test runs on the text channel; a hot text_temperature collapses every take to ~0.16 s of pad.
• audio_lm_heads.* / text_lm_head.weight reported MISSING at load is benign — those heads are weight-tied. Do not "fix" it, and never call initialize_local_text_lm_head_from_text_lm_head().

---

Training

Data — 1,288 clips (4.08 h), source-balanced:

The sampler draws each source with equal probability per epoch, so neither the synthetic English half nor the real German half can dominate the adapter. This is also why the adapters carry a German accent-flavoured energy that the pure-English base does not.

Hyperparameters: rank 16/32/64, alpha = 2 × rank, lora_dropout = 0.05, lr 2e-4 with linear decay, 8 epochs, bf16. Targets are the global q/k/v/o/gate/up/down projections, the local decoder's c_attn/c_proj/fc_in/fc_out, and all 12 audio_lm_heads — the audio heads matter; adapting attention alone moves the voice much less. All three ranks trained on identical batches against one shared frozen base, so the rank comparison is paired.

The data filter is the solid result from this line of work

Stronger than anything about the adapters themselves:

The literal WER × quality selection formula put 602 of 1,000 candidates at exactly score 0 (60 % had WER 0.00) and therefore selected the half with worse transcription. Replacing it with a listening filter kept 820 of 1,000 at the top rating, and a blind judge scored the corrected selection higher on 3 of 5 dimensions:

That result is properly powered and it stands. Never divide quality by (1 + WER) — most candidates have a negative core score, so the division form increases the reward as transcription gets worse.

---

Contents

real_r32_e2/  rank 32, epoch 2, real data only -- lowest WER, half the size
real_r32_e1/  rank 32, epoch 1, real data only
real_r64_e8/  rank 64, epoch 8, real data only -- THE DEFAULT (chosen by listening)
r64_e1/       rank 64, epoch 1, mixed data (first run)
r64_e8/       rank 64, epoch 8, mixed data (first run)
r32_e1/       rank 32, epoch 1, mixed data (first run)
samples/      12 MP3s: 3 held-out prompts x {base, r64_e1, r64_e8, r32_e1}
eval_prompts.json          the 16 held-out prompts, ready for build_user_message
eval_results.csv           13-cell table, mixed-data run
eval_results_real_only.csv 13-cell table, real-data-only run

Each adapter directory holds adapter_config.json + adapter_model.safetensors.

Caveats

• Judged by a model (gemini-3-flash), not by human raters.
• The evaluation is English-only, 16 prompts × 2 takes per cell. Training was mixed English/German; German output was not separately evaluated.
• The mined half is real broadcast audio; the generated half is synthetic. Anything the adapters reproduce about crowd noise or broadcast character comes from the mined half.
• No claim is made that these adapters improve on the base model at p < 0.0042. See the top of this card.

Provenance

Trained by LAION as part of the MOSS voice-acting line. Full experimental record: LAION-AI/laion-moss-local-1.5-voice-acting-4.55b.