Moonshine student m15s2, SAPC2 compression ladder (epoch 3, then RLOO)

Fine-tuned for the Speech Accessibility Project Challenge 2 (SAPC2): English speech from speakers with ALS, cerebral palsy, Down syndrome, Parkinson's disease and stroke.

Evaluation

The checkpoint was selected on a fixed 1,000-recording subset of the official Dev set; the scores below are on the full Dev set (47,929 recordings). Scores use the competition normalizer against the better of the two references; they are within about 0.1 points of the official SCTK scorer on the same predictions.

Full Dev (47,929 recordings)

Whole-recording decoding (200 s windows). Quick scorer: competition normalizer, better of the two references. Official: the challenge's evaluate.py with sclite against both references (where it was run; it scores about 0.05 points lower than the quick scorer).

decoding	WER	CER	official SCTK WER	official SCTK CER	empty transcripts	capped (ran to the token limit)
greedy	31.64%	20.24%	31.60%	20.20%	0	21
beam 4	28.37%	17.97%	28.31%	17.92%	0	33

Exact transcripts 44.30%; substitutions 23.30%, deletions 5.49%, insertions 3.65% of reference words (uncapped WER 32.45%).

CPU streaming package cost

The ONNX package run through the challenge's streaming interface on a 300-recording Dev subset (0.64 h of audio), two worker processes, one ONNX Runtime thread each, on a laptop CPU, with no other load. Compute is the wall-clock seconds spent per hour of audio; WER/CER are on that subset only.

final decode	compute per audio hour	WER	CER
greedy	246 s	31.48%	20.33%
beam 4	296 s	28.07%	17.47%

WER by duration

bucket	recordings	WER	share of all errors
1-2s	670	34.94%	0.37%
2-3s	4354	21.24%	2.32%
3-5s	14700	20.34%	9.97%
5-8s	14739	26.65%	18.57%
8-12s	7023	34.66%	16.53%
12-20s	3867	39.17%	16.78%
20-35s	1657	37.53%	15.68%
>35s	919	37.53%	19.79%

WER by etiology

etiology	recordings	WER	share of all errors
ALS	10232	29.01%	20.72%
Cerebral Palsy	11472	41.19%	24.99%
Down Syndrome	7719	35.99%	13.22%
Parkinson's Disease	14543	27.99%	36.43%
Stroke	3963	28.73%	4.64%

Runaway transcripts (hypothesis over 1.5x the reference length): 291, 2.96% of all errors; truncated transcripts: 367.

Streaming latency (CPU package)

Measured with the challenge's own local decoder and latency script on a 100-recording Dev subset (12.5 min of audio), audio sent at real-time pace in 100 ms chunks, one ONNX Runtime thread, on a laptop CPU. TTFT here is the first partial relative to the start of the audio (the official metric subtracts the MFA speech onset, which this subset does not carry); partial transcripts are greedy at every setting, so TTFT does not depend on the final beam width. TTLT is the final transcript after the last audio chunk.

final beams	TTFT median	TTLT median	TTLT p90	TTLT p95	TTLT max	full-Dev WER / CER
1	415 ms	130 ms	245 ms	290 ms	1729 ms	31.64% / 20.24%
4	416 ms	152 ms	359 ms	397 ms	4265 ms	28.37% / 17.97%

Training recipe

{
  "objective": "rl",
  "init": "/runs/sapc_moonshine/runs/m15s2_e3/best",
  "kd_weight": 0.0,
  "peak_lr": 2e-06,
  "epochs": 1,
  "balanced": true,
  "balance_max_repeats": 2,
  "rl": {
    "samples_per_recording": 4,
    "kl_weight": 0.02,
    "ce_anchor": 0.1,
    "logprob_reduction": "sum"
  },
  "policy": {
    "max_items": 96,
    "padded_seconds": 576,
    "attention_cost": 60000,
    "decoder_tokens": 12288,
    "accumulation": 2
  },
  "seed": 20261027
}

Optimizer: blocked SOAP (Adam in a Shampoo eigenbasis), FP32 weights with BF16 autocast. Code: the moonshine package in the project repository (training, distillation, RL and evaluation), with resumable checkpoints.

Files

Hugging Face format (config.json, model.safetensors, tokenizer/processor files). Load with transformers (MoonshineForConditionalGeneration or CohereAsrForConditionalGeneration).

vanshnawander/sapc2-moonshine-m15s