Moonshine Base, Welsh (compact)

A smaller and more accurate version of rebreda/moonshine-base-welsh, a Welsh speech recogniser fine-tuned from Moonshine Base. It has 53.2M parameters instead of 61.5M, and the weights are stored in 16-bit floats, so the download is 106 MB instead of 246 MB.

It was made from the earlier model in three steps, none of which involved more training:

1. Checkpoint averaging. The LoRA adapter is the average of ten checkpoints of the continuation run (steps 4,000 to 8,500), not a single checkpoint.
2. Five attention heads switched off. Tracing the model's repetition loops (the same word over and over) showed five decoder self-attention heads that copy the previous words and outbid the end-of-text token. They are heads 0, 1 and 3 of layer 4, head 1 of layer 2 and head 1 of layer 7 (counting from 0). Their output weights are set to zero (0.18% of the weights). The heads were found using development data and Banc validation clips only, never test data. On Siarad, the held-out conversation test, this cut WER from 87.0 to 76.4.
3. A smaller token table. The model kept the 32,768-token table of the original English model, and most of it was never used. This version keeps 12,735 tokens: those used at least 3 times in the training transcripts, all 256 byte tokens (so any word can still be spelled out), and the pieces the tokenizer needs to build them. The tokens were chosen from training text only. This removes 8.3M parameters.

The write-up is on the project page: https://quickthoughts.ca/projects/welsh-asr/.

• Parameters: 53.2M
• Languages: Welsh, and English words as spoken inside Welsh speech. Plain English got worse (see below).
• Input: 16 kHz mono audio, short utterances
• Output: text with casing and punctuation
• Licence: MIT (inherited from Moonshine)

How to use

The model works with plain transformers (tested with 5.20). Audio must be 16 kHz mono. The weights are stored in fp16 and load as fp32 by default; fp16 storage changed none of 600 test decodes.

import torch
import soundfile as sf
from transformers import AutoProcessor, MoonshineForConditionalGeneration

model_id = "rebreda/moonshine-base-welsh-compact"
processor = AutoProcessor.from_pretrained(model_id)
model = MoonshineForConditionalGeneration.from_pretrained(model_id).eval()

audio, sr = sf.read("clip.wav", dtype="float32")  # 16 kHz mono
assert sr == 16000

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
seconds = len(audio) / 16000
# Cap the output at about 25 tokens per second of audio (see below).
max_length = min(194, int(25 * seconds) + 10)

with torch.no_grad():
    ids = model.generate(**inputs, num_beams=5, max_length=max_length)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Or with the pipeline:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="rebreda/moonshine-base-welsh-compact")
print(asr("clip.wav", generate_kwargs={"num_beams": 5})["text"])

Decoding settings

• Beam search, 5 beams. All the scores below were measured this way. The transformers speech pipeline adds num_beams=5 by itself when the generation config does not set it, and this model's generation config does not. If you call generate() yourself, pass num_beams=5.
• Cap the output length at about 25 tokens per second of audio. The switched-off heads remove most repetition loops (on Siarad, 4 of 1,500 clips still loop), but a cap of 25 tokens per second plus 10 is a cheap safety net. Fast Welsh conversation reaches about 21 tokens per second at the extreme. The scores below were measured without the cap.
• Long recordings should be cut into short segments (for example with a voice activity detector) before transcription. The model was trained and tested on single utterances.

Results

Word error rate (WER, %, lower is better). Text was normalised with Whisper's basic normaliser, keeping diacritics. Beam 5, no length cap, no language model.

The mean over Common Voice, FLEURS Welsh, Arfor and Siarad is 40.1 (the earlier model: 43.5).

Where the gains come from, measured on the same test sets with paired bootstrap confidence intervals: checkpoint averaging improved Arfor by 2.1 points (95% CI 0.6 to 4.3); switching off the five heads improved Siarad by 10.6 points (5.7 to 16.3) and changed no other set by more than 0.5; the smaller token table changed every Welsh set by 0.4 points or less and cost 4.5 points (3.9 to 5.2) on English.

Please read these caveats before quoting the numbers:

• English got worse. The training text had only about 2,600 English sentences, so most English word tokens were removed. English words that are common inside Welsh speech were kept, and the Welsh sets did not get worse, but plain English went from 16.4 to 21.0 WER. If you need English, use rebreda/moonshine-base-welsh or Moonshine itself.
• Arfor: most Lleisiau Arfor test sentences also appear as prompts in Arfor's training split (different speakers, same words). The fair figure is the 33.8 on the 403 test clips whose sentence is not in Arfor's training data.
• Siarad is hard. It is natural conversation with frequent switching between Welsh and English, and it was never used for training or for choosing the edits. Even at 76.8, this model is not good enough for transcribing free conversation.
• Common Voice: the test set is Common Voice 27.0's Welsh test split. Training used other Common Voice 27.0 clips whose audio, speaker and sentence stay out of its dev and test splits.
• Bangor's Whisper is more accurate, but much larger. techiaith/whisper-large-ft-cy-en (Whisper large-v2, 1.5B parameters, about 29 times this model) scored 27.6 on unseen Arfor against 33.8 here. On the Banc podcast test, which both models trained on, it scored 34.7 and this model 32.7. On Siarad it scored 103.8. Its Common Voice score (10.3) may be inflated: it trained on an older Common Voice release whose clips may overlap the CV 27.0 test.

With an external language model

Not part of this repository, but for context: rescoring 8-beam candidate lists of the same model before the token-table cut with a Welsh word n-gram model and Bangor's Welsh Llama 3.2 1B gave Common Voice 20.5, FLEURS 34.6, Arfor 18.8, Siarad 71.7 and Banc 31.8 (mean 36.4).

Training

Training used listenr on one AMD Radeon 8060S (ROCm, bf16), in two stages:

1. From moonshine-base: a rank-128 LoRA (alpha 256, learning rate 3e-4) on Common Voice 27.0 Welsh, Lleisiau Arfor's Welsh train rows and FLEURS English train (about 60 hours), then merged into the weights.
2. Continuation: a fresh rank-128 LoRA (alpha 256, learning rate 1e-4, warm-up 200 steps, batch 32) on the same data plus Bangor University's Banc Trawsgrifiadau Bangor and Corpws CLlLC: 129,938 clips, 175.1 hours. The adapter weights of the ten checkpoints from steps 4,000 to 8,500 were averaged and then merged into the model. The average was chosen over single checkpoints on 3,000 dev clips.

Then the three edits above were made directly to the weights. Training text was cleaned: Bangor's markup for non-speech sounds and English words was removed, and {spoken|written} pairs were reduced to the spoken form. Lleisiau Arfor rows were filtered to Welsh.

Training audio (175.1 h): Common Voice 23.9 h, Lleisiau Arfor 28.4 h, FLEURS English 7.5 h, Banc 63.0 h, CLlLC 52.4 h.

Training data and licences

FLEURS is by Conneau et al., Google (CC-BY 4.0): "FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech" (2022). Banc Trawsgrifiadau Bangor and Corpws CLlLC are published by the Language Technologies Unit (Uned Technolegau Iaith), Bangor University. Banc's terms include an agreement not to try to identify its speakers; please respect it. Common Voice's terms also forbid trying to identify speakers.

Intended use and limitations

• Good for: short, fairly clear Welsh utterances, such as dictation, voice commands, read speech and podcast-style speech.
• Not good for: free conversation with a lot of code-switching (see Siarad), plain English, noisy audio, or anywhere errors have consequences. About one word in four is still wrong on read speech.
• Like the base model, please do not use it to transcribe recordings of people made without their consent.

Credits and licence

Based on Moonshine by Useful Sensors / Moonshine AI, released under the MIT licence:

MIT License

Copyright (c) 2024 Useful Sensors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Please cite Moonshine if you use this model:

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

The fine-tuned weights in this repository are released under the same MIT licence. Thanks to Mozilla Common Voice and its Welsh contributors, the Lleisiau Arfor project, Google (FLEURS) and Bangor University's Language Technologies Unit for the data.