SOTA-MOS: speech MOS at 16, 24 and 48 kHz

SOTA-MOS predicts the mean opinion score (1–5) of synthetic speech as listeners rate it in a test that mixes sampling rates. It is built for AudioMOS Challenge 2025 Track 3. Code, training and every experiment: Scicom-AI-Enterprise-Organization/SOTA-MOS.

It beats every Track 3 system, including the winner HighRateMOS, on 7 of 8 official metrics: utt MSE, utt LCC, utt SRCC, utt KTAU, sys LCC, sys SRCC, sys KTAU.

Eval labels were never used to build or select the model. Members were chosen by greedy forward selection on out-of-fold predictions of the dev set, by a rule fixed before any of our systems was scored on eval. The eval set was scored once.

SOTA-MOS keeps the order of the 20 conditions across sampling rates. The replicated HighRateMOS puts every good condition near 3.6. Ours spreads them out, and it puts 16 kHz audio below the same system at 24 or 48 kHz, as the listeners did.

Use

git clone https://github.com/Scicom-AI-Enterprise-Organization/SOTA-MOS && cd SOTA-MOS
uv sync --extra serve
hf download Scicom-intl/SOTA-MOS --include "model/*" --exclude "model/onnx/*" --local-dir .
uv run python -m sotamos.predict --system model/system.json clip_16k.wav clip_28k.wav clip_44k.wav clip_48k.wav clip_48k_world.wav

The same natural sentence at four rates, and a WORLD-vocoded copy at 48 kHz:

file,sampling_rate,mos
clip_16k.wav,16000,3.7021
clip_28k.wav,24000,3.9625
clip_44k.wav,48000,3.7948
clip_48k.wav,48000,3.8229
clip_48k_world.wav,48000,2.3365

From Python:

from sotamos.predict import SystemScorer, load_audio

scorer = SystemScorer("model/system.json", "cuda")
x, sr, x16 = load_audio("clip.wav")   # any rate: resampled to the nearest of 16 / 24 / 48 kHz
print(scorer(x, sr, x16))              # MOS on the mixed-rate listening-test scale

CPU, without PyTorch (ONNX)

model/onnx/ holds the same ensemble as ONNX graphs, one per member: the frozen backbone runs once and feeds its 5 fold heads. It needs only onnxruntime, numpy and soundfile.

hf download Scicom-intl/SOTA-MOS --include "model/onnx/*" --local-dir .
pip install onnxruntime numpy soundfile
python model/onnx/onnx_predict.py --model model/onnx clip_44k.wav clip_16k.wav
python model/onnx/onnx_predict.py --model model/onnx --int8 --threads 4 clip_44k.wav

Sampling rates

16, 24 and 48 kHz are scored as they are. Any other rate goes to the nearest of the three:

Serve

A FastAPI server with dynamic batching. It follows faster-UTMOSv2/serving, with batches built by clip length because SOTA-MOS scores the whole clip.

cd serving
SYSTEM=../model/system.json MAX_BATCH=16 PP_WORKERS=16 bash run_serve.sh
curl -X POST http://127.0.0.1:8000/predict -F file=@clip_44k.wav
# {"mos": 3.7948, "reps": 1, "gpu_ms": 141.17, "total_ms": 161.6, "batch_size": 1,
#  "input_sampling_rate": 44100, "sampling_rate": 48000, "duration_s": 5.599}

One GPU, the 400 Track 3 eval clips (1,506 s of audio), same client for both servers, run back to back:

SOTA-MOS serves as fast as faster-UTMOSv2 and holds up better at mid load. faster-UTMOSv2 scores one fold on a 3-second crop. SOTA-MOS scores the whole clip with 3 models × 5 folds. Served scores match the offline scores within 0.007 MOS on average (Pearson r ≥ 0.9998 at batch 1, 8 and 32).

The server is a drop-in replacement for faster-UTMOSv2's server. Same routes, same file field, same ?reps= and ?dataset= parameters (accepted; SOTA-MOS is deterministic and has no data-domain), same response keys in the same order and types, same status codes. It adds three keys after them: input_sampling_rate, sampling_rate (the trained rate the clip was scored at) and duration_s.

SOTA-MOS:       {"mos": 3.7948, "reps": 1, "gpu_ms": 141.17, "total_ms": 161.6, "batch_size": 1, "input_sampling_rate": 44100, "sampling_rate": 48000, "duration_s": 5.599}
faster-UTMOSv2: {"mos": 3.21875, "reps": 1, "gpu_ms": 130.26, "total_ms": 150.69, "batch_size": 1}

serving/check_api_compat.py runs both servers side by side and checks every route: POST /predict (with and without reps/dataset), invalid reps (422), a missing file (422), non-audio bytes (500), /health, /stats, /warmup and /. All pass. faster-UTMOSv2's own benchmark.py runs against the SOTA-MOS server unchanged.

Model

Three frozen self-supervised speech models, each followed by a small trained head, 5 cross-validation folds each:

Each head combines the backbone's early layers (weighted sum over the kept layers), a multi-scale CNN on the clip's native-rate mel spectrogram on a 0–24 kHz axis, and learned embeddings for the sampling rate and the listening test. The mel branch is what lets a 16 kHz clip look different from a 48 kHz one. Training used the Track 3 train labels (single-rate tests) and the dev labels (mixed-rate test), with the listening test as an input.

The mixed-rate test drops 16 kHz audio by 0.35 MOS against its single-rate test. Train labels alone cannot teach that. The replication, the ablations and the frozen-feature probes are in the GitHub README.

Files

References

• W. Ren et al., HighRateMOS: Sampling-Rate Aware Modeling for Speech Quality Assessment, 2025.
• W.-C. Huang et al., The AudioMOS Challenge 2025, 2025.
• K. Baba et al., UTMOSv2, 2024; faster-UTMOSv2.