sea-asr-ms — Malay speech recognition on EmbeddingGemma 2 (CTC)

Google's EmbeddingGemma 2 audio encoder with a CTC head, trained by Pixel ML on openly licensed Malay speech: YODAS and YODAS2 (ESPnet/CMU, CC-BY YouTube audio), VoxLingua107 (TalTech, CC-BY-4.0) and FLEURS train (Google, CC-BY-4.0), 79.5 h in total. An optional n-gram language model (folder lm/, 0.40 GB) built from Malay Wikipedia, FineWeb-2 Malay web text and our transcripts comes with the weights.

Speech to text for Malay. 364M parameters. On FLEURS, with its n-gram LM: on par with Qwen3-ASR-1.7B (6× larger) and 7 WER points better than Qwen3-ASR-0.6B. Whisper large-v3 and turbo are still more accurate on this set.

FLEURS is read speech only; we have no Malay conversational test set yet. Whisper and Qwen3-ASR may have trained on FLEURS, so their FLEURS rows may not be held-out scores. Our model trained on FLEURS ms_my train; the test and validation sentences were held out of all our training data and of the LM text.

Labels: PixelML/sea-asr-ms-data (rows only, no audio). Code: github.com/PixelML/sea-asr.

Results

FLEURS ms_my test (749 utterances, read speech). One normaliser for all systems (quanphid/text.py, "ms": lower case, no punctuation, numbers as Malay words). WER % (lower is better), 95 % bootstrap CI (1,000 resamples). Paired diff = sea-asr-ms + LM minus the rival on the same resamples; a CI that excludes 0 is a real difference. Params = Hugging Face safetensors total.

• LM settings (alpha 0.5, beta 2.0, beam 128) and the choice among 3 candidate LMs were made on FLEURS ms_my validation only. The LM text has every sentence removed that equals or shares an 8-gram with a FLEURS ms_my validation or test reference.
• Ship check: this folder, loaded with the public sea_asr v0.2.1 package, gives greedy 13.71 and + LM 8.77 on the same test set on an H100 (training-time evaluation: 13.73 and 8.83).
• Rescoring (opt-in): a second pass reranks the LM beam search's 32 best hypotheses with Sea AI Lab's Sailor2-1B (Apache-2.0, pinned commit), with a Malay prompt and weights a 0.4, b 0.4, c 2.0, chosen on FLEURS ms_my validation only and stored in lm/lm_config.json ("rescore" block, read by sea_asr v0.2.2). Rescoring minus rival: Whisper large-v3 +1.26 [+0.61, +1.93] (Whisper better), Whisper large-v3-turbo +0.43 [−0.25, +1.10] (CI includes 0), Qwen3-ASR-1.7B −1.14 [−1.84, −0.45] (ours better). Whisper is more accurate. With sea_asr v0.2.1 the block is ignored and rescore=True uses the Indonesian defaults (8.54 on this test set).
• Ship check, rescoring: this folder with sea_asr v0.2.2 and rescore=True (settings read from lm/lm_config.json, no arguments) gives 8.35 on the same test set on an H100; 749 / 749 texts equal the tuning run.

Speed

One H100 80GB, FLEURS ms_my test (749 short clips, 2.26 h), side by side with Whisper on the same GPU type and container size (32 vCPU). Speed = audio seconds / wall seconds of the whole transcribe() call, model load and warm-up excluded; ours sea_asr v0.2.1 with this folder, the beam search on CPU (one worker, the default).

Greedy is about 30× faster than Whisper large-v3 and 18× faster than turbo on short clips; with the LM, 3.0× and 1.8×. On long-form audio the LM beam search costs more. Long-form check, same H100 type and container size: one 57-minute Malay talk show held out of training (Media Selangor, "Bicara Semasa", 7 April 2026, CC BY, youtube.com/watch?v=srRfnfRZM2Q), whole file in one transcribe() call, model load and warm-up excluded, median of 3 runs (ours) and 2 runs (Whisper):

On this clip greedy is about 24× faster than Whisper large-v3; with the n-gram LM it is faster than large-v3 and a little slower than turbo. Whisper here: faster-whisper 1.2.1 batched long-form pipeline (fp16, Silero VAD, batch 16, beam 5). Whisper setup: faster-whisper 1.2.1 fp16, beam 5, Silero VAD, batch 16 clips. Whisper spends part of its wall time on per-clip CPU work, and batch 16 is conservative for an H100; a Whisper setup tuned for throughput would narrow the gap. Whisper's WER in this run differs from the table above (different decode setup); it is more accurate in both.

Use

pip install "sea-asr[lm] @ git+https://github.com/PixelML/sea-asr@v0.2.2"   # drop [lm] for greedy only
pip install "sea-asr[rescore] @ git+https://github.com/PixelML/sea-asr@v0.2.2"   # + opt-in Sailor2-1B rescoring

from sea_asr import SeaASR, to_srt

asr = SeaASR.from_pretrained("PixelML/sea-asr-ms")                            # greedy; device="cuda" or "cpu"
asr = SeaASR.from_pretrained("PixelML/sea-asr-ms", lm="PixelML/sea-asr-ms")   # + n-gram LM from lm/ (recommended)
asr = SeaASR.from_pretrained("PixelML/sea-asr-ms", lm="PixelML/sea-asr-ms", rescore=True)  # + rescoring (GPU advised)
print(asr.transcribe("clip.wav"))

out = asr.transcribe("berita.wav", return_words=True)    # {"text", "words": [{"word", "start", "end"}]}
open("berita.srt", "w").write(to_srt(out["words"]))

1
00:00:01,320 --> 00:00:03,480
beliau tidak dapat mengambil
ubat yang diperlukan untuk

2
00:00:03,680 --> 00:00:06,760
menahan sakit kerana ia telah
diharamkan dari perlawanan

Cues: at most 2 lines × 32 characters, at most 6 s. Input: 16 kHz mono (other rates are resampled). Inputs longer than 30 s are cut at quiet points; timestamps stay on the original timeline. CLI: python -m sea_asr --model PixelML/sea-asr-ms --lm PixelML/sea-asr-ms clip.wav --srt.

Limits

• Tested on FLEURS only (read speech). Accuracy on conversational, broadcast or noisy Malay is not measured.
• Lower case, no punctuation; caption-ready output needs a punctuation + casing restorer.
• Without the LM the model is much less accurate (13.7 vs 8.8 WER on FLEURS); use lm= unless speed matters most.
• Machine labels from Qwen3-ASR-1.7B and Whisper large-v3: the model inherits their shared errors.
• Malay and Indonesian are close; some training clips may be Indonesian (VoxLingua107 labels come from video titles, and the Indonesian-marker filter is not perfect). Code-switching to English is common in the data and not evaluated. Speaker, accent, dialect and gender balance are not measured.
• Do not use for speaker identification or surveillance.

Training

Initialised from our Indonesian model chain: EmbeddingGemma 2 encoder + CTC head trained for Indonesian (PixelML/sea-asr-id), then a first Malay round (YODAS ms000 + FLEURS train + an equal-hours Indonesian YODAS slice), then this round on Malay only (79.5 h, 1,200 steps, best checkpoint at step 800 on FLEURS ms_my validation), on one NVIDIA DGX Spark (GB10).

Consensus labels: Qwen3-ASR-1.7B and Whisper large-v3 transcribe each clip; we keep the Qwen3-ASR text where the two agree (character error rate ≤ 0.2), with no repeat loop and not English; rows that look Indonesian are removed. The labels are in PixelML/sea-asr-ms-data. Datasets without a licence were not used.

The n-gram LM: KenLM 4-gram on Malay Wikipedia (dump 20231101, 57.1 M words), FineWeb-2 zsm_Latn (200 M words, first file, deduplicated) and our Malay transcripts (0.16 M words), after the test filter; pruned, 0.40 GB. Build details in lm/manifest.json.

Credits

• Base encoder: Google DeepMind, EmbeddingGemma 2 (914f7f89)
• Data: YODAS and YODAS2 (ESPnet/CMU, CC-BY-3.0; YODAS 52c5a1b9, YODAS2 c9674490), subset ms000; CC-BY YouTube creators listed in ATTRIBUTION.tsv
• Data: VoxLingua107 by Jörgen Valk and Tanel Alumäe, Tallinn University of Technology (TalTech), CC-BY-4.0 (TalTechNLP/voxlingua107_wds 8dfcfa5b)
• Data and test set: FLEURS ms_my train and test (Google, google/fleurs 70bb2e84, CC-BY-4.0)
• Teachers: Qwen3-ASR-1.7B (Alibaba Qwen, 7278e1e7, Apache-2.0); Whisper large-v3 (OpenAI, MIT) via faster-whisper (SYSTRAN)
• Language model text: Malay Wikipedia, Wikipedia contributors / Wikimedia Foundation (wikimedia/wikipedia 20231101.ms, b04c8d1c, CC-BY-SA-3.0 + GFDL); FineWeb-2, Hugging Face (HuggingFaceFW/fineweb-2 zsm_Latn, af9c1333, ODC-By-1.0)
• Language model tools: KenLM by Kenneth Heafield (kpu/kenlm 4cb443e6); pyctcdecode 0.5.0 (Kensho Technologies, kensho-technologies/pyctcdecode)
• Rescoring LM (opt-in): Sailor2-1B by Sea AI Lab (sail/Sailor2-1B 75c2856f, Apache-2.0)
• Long-form speed clip: Media Selangor, "Bicara Semasa" (7 April 2026, CC BY, youtube.com/watch?v=srRfnfRZM2Q); timing only, not in training
• Recipe: Quan Nguyen (@qnguyen3), QUAN-Transcribe — https://github.com/qnguyen3/QUAN-Transcribe (4f45e02)

Licence

Model weights and code: Apache-2.0. Base encoder EmbeddingGemma 2 is Apache-2.0; deployments must adhere to the Gemma Prohibited Use Policy.

The n-gram LM in lm/ is derived from Wikipedia text (CC BY-SA 3.0) and FineWeb-2 (ODC-By 1.0) and is licensed CC BY-SA 3.0 (see lm/LICENSE.md). If you ship or modify lm/, keep the attribution and share it under the same licence. Greedy decoding does not use lm/.