Fifth generation of the surgical emotion module (v1: amir0907/student · v2: amir0907/student-v2 · v3: amir0907/student-v3 · v4: amir0907/student-v4).
Same interface as v4 (frozen emotion2vec_plus_base backbone + 3.18 M-param adapter, 6.25 Hz = exactly 1 emotion frame per dots.tts LLM token, salience-gated absolute-energy intensity target, corr+std losses). v5 adds the one thing v4 proved was missing: the label-free intensity proxy had a hard ceiling (the target itself separated RAVDESS normal/strong by only Cohen-d 0.23, and the v4 student sat at 0.26). v5 breaks through it with weak supervision from the intensity labels that already existed in the training corpora but were never used:
01=normal, 02=strong → 0.0 / 1.0LO=0.0, MD=0.5, HI=1.0 (XX skipped)Pairwise margin ranking on the utterance-mean intensity (mean of the 6.25 Hz
frame intensities): for every labeled pair with label_i > label_j,
loss = relu(margin − (pred_i − pred_j)), margin 0.10. The term is
shift-invariant, so it cannot fight the frame-level distillation target — it only
enforces ORDERING (strong > normal, HI > MD > LO). Every 2nd batch is drawn from
the labeled subset so pairs are frequent. All other losses/recipe identical to v4;
same cache; 4000 AdamW steps, batch 8, fp16, single T4 (~14 min).
| metric | v1 | v2 | v4 | v5 |
|---|---|---|---|---|
| top-1 agreement | 100 % | 100 % | 100 % | 100 % |
| mean JS divergence ↓ | 0.00035 | 0.00022 | 0.00044 | 0.00067 |
| frame-level cosine (up to 50 Hz) | 0.99524 | 0.99572 | 0.99198 | 0.99294 |
| intensity corr (new target) | 0.237 | 0.239 | 0.594 | 0.629 |
| intensity curve std (dynamics) | 0.030 | 0.035 | 0.183 | 0.171 |
| normal(01) vs strong(02) mean-intensity diff | +0.021 | +0.027 | +0.047 | +0.063 |
| intensity Cohen-d (336 emotional files) | 0.57 | 0.66 | 0.26 | 0.34 |
| target's own Cohen-d (ceiling) | — | — | 0.23 | 0.34 — ceiling exceeded |
| CREMA-D LO/MD/HI Spearman (in-train) | 0.30 | 0.33 | 0.29 | 0.32 |
| per-emotion strong>normal | — | — | — | 5/7 |
Rate contract exact at 6.25 Hz for 1/2.5/5/10 s inputs.
Authoritative numbers: reports/v1_v2_v4_v5_compare.json, reports/eval_report_v5.json.
Honest notes:
demo/)9 samples: one strong-intensity utterance per emotion (03-08) + neutral + the funasr zh clip — each with the actual test audio, the v5 analysis plot and numeric results. RAVDESS audio is CC BY-NC-SA 4.0 (Livingstone & Russo 2018) — non-commercial use only; the zh clip is MIT.
from code.surgical_emotion import SurgicalEmotionEncoder
enc = SurgicalEmotionEncoder.from_pretrained(
"/path/to/emotion2vec_plus_base", # frozen teacher (funasr dir, Apache-2.0)
"surgical_emotion_module.pt", # this repo's checkpoint (G=8 restored automatically)
device="cuda:0")
out = enc(wav_16k, wav_lengths)
# frame_embs (B,T,768) @6.25Hz | frame_probs (B,T,9) | frame_intensity (B,T,1)
# utt_emb (B,768) | utt_probs (B,9) | frame_lengths (B,)
# T = ceil(duration_s * 6.25); frame i covers audio [i*0.16, (i+1)*0.16) s
frame_embs are designed for per-token time-varying
conditioning of dots.tts (1 emotion frame per 6.25 Hz LLM token).