Moonshine Base, Welsh

A Welsh speech recogniser fine-tuned from Moonshine Base. It is a small model: it runs on a CPU and is about 29 times smaller than the leading OS Welsh Whisper models from Bangor University. Learn about how the model was trained at the project website.

Versions in this repository

Each version is a branch. Pick one with revision=.

Welsh mean = mean WER over Common Voice, FLEURS Welsh, Lleisiau Arfor and Siarad. Use main for Welsh. Use full-vocab if you also need plain English. v1 is kept so earlier results can be reproduced.

model = MoonshineForConditionalGeneration.from_pretrained(
    "rebreda/moonshine-base-welsh", revision="full-vocab")

main and full-vocab were made from v1's training run with no further training. The steps were:

1. Checkpoint averaging. The LoRA adapter is the average of ten checkpoints of the continuation run (steps 4,000 to 8,500), not a single checkpoint.
2. Five attention heads switched off. Tracing the model's repetition loops (the same word over and over) showed five decoder self-attention heads that copy the previous words and outbid the end-of-text token. They are heads 0, 1 and 3 of layer 4, head 1 of layer 2 and head 1 of layer 7 (counting from 0). Their output weights are set to zero (0.18% of the weights). The heads were found using development data and Banc validation clips only, never test data. On Siarad, the held-out conversation test, this cut WER from 87.0 to 76.4.
3. A smaller token table (main only). The model kept the 32,768-token table of the original English model, and most of it was never used. main keeps 12,735 tokens: those used at least 3 times in the training transcripts, all 256 byte tokens (so any word can still be spelled out), and the pieces the tokenizer needs to build them. The tokens were chosen from training text only. This removes 8.3M parameters.

The write-up is on the project page: https://quickthoughts.ca/projects/welsh-asr/.

• Parameters: 53.2M (main), 61.5M (full-vocab)
• Languages: Welsh, and English words as spoken inside Welsh speech. Plain English is worse on main (see below).
• Input: 16 kHz mono audio, short utterances
• Output: text with casing and punctuation
• Licence: MIT (inherited from Moonshine)

How to use

The model works with plain transformers (tested with 5.20). Audio must be 16 kHz mono. The weights are stored in fp16 and load as fp32 by default; fp16 storage changed none of 600 test decodes.

import torch
import soundfile as sf
from transformers import AutoProcessor, MoonshineForConditionalGeneration

model_id = "rebreda/moonshine-base-welsh"
processor = AutoProcessor.from_pretrained(model_id)
model = MoonshineForConditionalGeneration.from_pretrained(model_id).eval()

audio, sr = sf.read("clip.wav", dtype="float32")  # 16 kHz mono
assert sr == 16000

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
seconds = len(audio) / 16000
# Cap the output at about 25 tokens per second of audio (see below).
max_length = min(194, int(25 * seconds) + 10)

with torch.no_grad():
    ids = model.generate(**inputs, num_beams=5, max_length=max_length)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Or with the pipeline:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="rebreda/moonshine-base-welsh")
print(asr("clip.wav", generate_kwargs={"num_beams": 5})["text"])

For another version, add revision="full-vocab" to both from_pretrained calls, or to pipeline(...).

Decoding settings

• Beam search, 5 beams. All the scores below were measured this way. The transformers speech pipeline adds num_beams=5 by itself when the generation config does not set it, and this model's generation config does not. If you call generate() yourself, pass num_beams=5.
• Cap the output length at about 25 tokens per second of audio. The switched-off heads remove most repetition loops (on Siarad, 4 of 1,500 clips still loop), but a cap of 25 tokens per second plus 10 is a cheap safety net. Fast Welsh conversation reaches about 21 tokens per second at the extreme. The scores below were measured without the cap.
• Long recordings should be cut into short segments (for example with a voice activity detector) before transcription. The model was trained and tested on single utterances.

Results

Word error rate (WER, %, lower is better; CER in brackets). Text was normalised with Whisper's basic normaliser, keeping diacritics. Beam 5, no length cap, no language model.

Where the gains come from, measured on the same test sets with paired bootstrap confidence intervals: checkpoint averaging improved Arfor by 2.1 points (95% CI 0.6 to 4.3); switching off the five heads improved Siarad by 10.6 points (5.7 to 16.3) and changed no other set by more than 0.5; the smaller token table changed every Welsh set by 0.4 points or less and cost 4.5 points (3.9 to 5.2) on English.

Please read these caveats before quoting the numbers:

• English got worse. The training text had only about 2,600 English sentences, so most English word tokens were removed. English words that are common inside Welsh speech were kept, and the Welsh sets did not get worse, but plain English went from 16.4 to 21.0 WER. If you need English, use the full-vocab branch.
• Arfor: most Lleisiau Arfor test sentences also appear as prompts in Arfor's training split (different speakers, same words). The fair figure is the 33.8 on the 403 test clips whose sentence is not in Arfor's training data.
• Common Voice: the test set is Common Voice 27.0's Welsh test split. Training used other Common Voice 27.0 clips whose audio, speaker and sentence stay out of its dev and test splits.

Training

Training used listenr on one AMD Radeon 8060S (ROCm, bf16), in two stages:

1. From moonshine-base: a rank-128 LoRA (alpha 256, learning rate 3e-4) on Common Voice 27.0 Welsh, Lleisiau Arfor's Welsh train rows and FLEURS English train (about 60 hours), then merged into the weights.
2. Continuation: a fresh rank-128 LoRA (alpha 256, learning rate 1e-4, warm-up 200 steps, batch 32) on the same data plus Bangor University's Banc Trawsgrifiadau Bangor and Corpws CLlLC: 129,938 clips, 175.1 hours. The adapter weights of the ten checkpoints from steps 4,000 to 8,500 were averaged and then merged into the model. The average was chosen over single checkpoints on 3,000 dev clips.

Then the three edits above were made directly to the weights. Training text was cleaned: Bangor's markup for non-speech sounds and English words was removed, and {spoken|written} pairs were reduced to the spoken form. Lleisiau Arfor rows were filtered to Welsh.

Training audio (175.1 h): Common Voice 23.9 h, Lleisiau Arfor 28.4 h, FLEURS English 7.5 h, Banc 63.0 h, CLlLC 52.4 h.

Training data and licences

FLEURS is by Conneau et al., Google (CC-BY 4.0): "FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech" (2022). Banc Trawsgrifiadau Bangor and Corpws CLlLC are published by the Language Technologies Unit (Uned Technolegau Iaith), Bangor University. Banc's terms include an agreement not to try to identify its speakers; please respect it. Common Voice's terms also forbid trying to identify speakers.

Intended use and limitations

• Good for: short, fairly clear Welsh utterances, such as dictation, voice commands, read speech and podcast-style speech.
• Not good for: free conversation with a lot of code-switching (see Siarad), plain English on main, noisy audio, or anywhere errors have consequences. About one word in four is still wrong on read speech.
• Like the base model, please do not use it to transcribe recordings of people made without their consent.

Citation

If you use this model, please cite it and the Moonshine paper. Please also say which branch (main, full-vocab or v1) you used, since their scores differ.

@misc{rebreda2026moonshinewelsh,
      title={Moonshine Base, Welsh: a compact Welsh speech recogniser},
      author={rebreda},
      year={2026},
      howpublished={\url{https://huggingface.co/rebreda/moonshine-base-welsh}},
      note={Write-up: \url{https://quickthoughts.ca/projects/welsh-asr/}},
}

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

If your work depends on the training data, please also cite the datasets as their own cards ask.

Credits and licence

Based on Moonshine by Useful Sensors / Moonshine AI, released under the MIT licence:

MIT License

Copyright (c) 2024 Useful Sensors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

The fine-tuned weights in this repository are released under the same MIT licence. Thanks to Mozilla Common Voice and its Welsh contributors, the Lleisiau Arfor project, Google (FLEURS) and Bangor University's Language Technologies Unit for the data.