Moonshine Tiny, Welsh (pruned vocabulary)

A very small Welsh speech recogniser (with some English): 21.2M parameters. It was made from Moonshine Tiny by first cutting its vocabulary down to the tokens Welsh text needs, then fine-tuning all of its weights on 175 hours of Welsh and English speech.

It is less accurate than its bigger sibling, rebreda/moonshine-base-welsh (61.5M parameters), but a third of the size. The write-up of the experiments is on the project page: https://quickthoughts.ca/projects/welsh-asr/.

• Parameters: 21.2M (moonshine-tiny has 27.1M)
• Vocabulary: 12,160 tokens (moonshine-tiny has 32,768)
• Languages: Welsh, and English as spoken inside Welsh speech
• Input: 16 kHz mono audio, short utterances
• Output: text with casing and punctuation
• Licence: MIT (inherited from Moonshine)

The pruned vocabulary

Moonshine shares one table between its input embeddings and its output layer, so the 32,768-token vocabulary is about a third of moonshine-tiny's parameters. Welsh text uses only a fraction of those tokens. This model keeps 12,160 of them: every token the original tokenizer produces on a large body of Welsh text, the special and byte-fallback tokens (at their original ids), the tokenizer's added tokens, and the intermediate tokens BPE needs to build them. The embedding rows were cut to match, and the tokenizer in this repository is the pruned one.

Before fine-tuning, the pruned tokenizer produced exactly the same tokens as the original on all 204,116 Welsh texts checked, and the pruned model gave the same output scores on the kept tokens.

Nothing extra is needed to use it: the config, weights and tokenizer here already agree on the 12,160-token vocabulary, and it loads and decodes with plain transformers. Text outside the kept set (other scripts, rare English words) can still be written through byte-fallback tokens, just less efficiently.

Note: the Welsh text used to choose the kept tokens included the transcripts of some test sets (Lleisiau Arfor test, FLEURS Welsh test and Siarad), as well as Common Voice Welsh (all splits) and Lleisiau Arfor train. This only decides which tokens exist; the model never trained on the test audio or transcripts.

How to use

The model works with plain transformers (tested with 5.20). Audio must be 16 kHz mono.

import torch
import soundfile as sf
from transformers import AutoProcessor, MoonshineForConditionalGeneration

model_id = "rebreda/moonshine-tiny-welsh-pruned"
processor = AutoProcessor.from_pretrained(model_id)
model = MoonshineForConditionalGeneration.from_pretrained(model_id).eval()

audio, sr = sf.read("clip.wav", dtype="float32")  # 16 kHz mono
assert sr == 16000

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
seconds = len(audio) / 16000
# Cap the output at about 25 tokens per second of audio (see below).
max_length = min(194, int(25 * seconds) + 10)

with torch.no_grad():
    ids = model.generate(**inputs, max_length=max_length)  # greedy, as scored
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Or with the pipeline:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="rebreda/moonshine-tiny-welsh-pruned")
print(asr("clip.wav")["text"])

Decoding settings

• Greedy decoding. The scores below were measured with greedy decoding (one beam). This model's generation config sets num_beams: 1, so the transformers speech pipeline decodes greedily too. (The pipeline switches to 5 beams only when the generation config leaves num_beams unset, which is the case for the base-size model.) Beam search (num_beams=5) works but has not been scored on the test sets for this model.
• Cap the output length at about 25 tokens per second of audio. On conversational audio, decoding sometimes falls into a repetition loop (the same word over and over) that runs to the maximum length of 194 tokens. Capping each clip at 25 tokens per second plus 10 stops this without cutting real speech: fast Welsh conversation reaches about 21 tokens per second at the extreme. (The cap was measured on the base-size model, where it brought Siarad WER from 88.3 to 81.9 without hurting the other test sets.) The scores below were measured without the cap.
• Long recordings should be cut into short segments (for example with a voice activity detector) before transcription. The model was trained and tested on single utterances.

Results

Word error rate (WER, %, lower is better). Text was normalised with Whisper's basic normaliser, keeping diacritics. Greedy decoding, no length cap, no language model.

The mean over Common Voice, FLEURS Welsh, Arfor and Siarad is 53.7. For comparison, the 61.5M base-size model scores 43.5 on the same mean (with 5 beams).

Please read these caveats before quoting the numbers:

• Arfor: most Lleisiau Arfor test sentences also appear as prompts in Arfor's training split (different speakers, same words), so the full Arfor score is optimistic. The fairer figure is the 45.8 on the 403 test clips whose sentence is not in Arfor's training data.
• Siarad is hard. It is natural conversation with frequent switching between Welsh and English, and it was never used for training. At 92.7 this model is not usable for free conversation; part of that score is repetition loops.
• Common Voice: the test set is Common Voice 27.0's Welsh test split. Training used other Common Voice 27.0 clips whose audio, speaker and sentence stay out of its dev and test splits.
• Bangor's Whisper is far more accurate, but about 70 times larger: techiaith/whisper-large-ft-cy-en (Whisper large-v2, 1.5B parameters) scored 27.6 on unseen Arfor and 34.7 on the Banc podcast test.

Training

Training used listenr on one AMD Radeon 8060S (ROCm, bf16). Starting from the vocabulary-pruned moonshine-tiny, all weights were trained (full fine-tune, learning rate 1e-4, warm-up 500 steps, batch 32) on 129,938 clips, 175.1 hours. Early stopping on dev WER ended the run at step 12,500 (about 3 passes over the data); this model is the best checkpoint, step 12,000. The dev set is Common Voice dev plus Lleisiau Arfor dev.

Training text was cleaned: Bangor's markup for non-speech sounds and English words was removed, and {spoken|written} pairs were reduced to the spoken form. Lleisiau Arfor rows were filtered to Welsh.

Training audio (175.1 h): Common Voice 23.9 h, Lleisiau Arfor 28.4 h, FLEURS English 7.5 h, Banc 63.0 h, CLlLC 52.4 h.

Training data and licences

FLEURS is by Conneau et al., Google (CC-BY 4.0): "FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech" (2022). Banc Trawsgrifiadau Bangor and Corpws CLlLC are published by the Language Technologies Unit (Uned Technolegau Iaith), Bangor University. Banc's terms include an agreement not to try to identify its speakers; please respect it. Common Voice's terms also forbid trying to identify speakers.

Intended use and limitations

• Good for: short, clear Welsh utterances on very small devices, such as voice commands or dictation, where size matters more than accuracy.
• Not good for: free conversation (see Siarad), noisy audio, or anywhere errors have consequences. About one word in three is still wrong on read speech.
• English: it recognises some English (30.3 WER on FLEURS English), but it is tuned for Welsh.
• Like the base model, please do not use it to transcribe recordings of people made without their consent.

Credits and licence

Based on Moonshine by Useful Sensors / Moonshine AI, released under the MIT licence:

MIT License

Copyright (c) 2024 Useful Sensors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Please cite Moonshine if you use this model:

@misc{jeffries2024moonshinespeechrecognitionlive,
      title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
      author={Nat Jeffries and Evan King and Manjunath Kudlur and Guy Nicholson and James Wang and Pete Warden},
      year={2024},
      eprint={2410.15608},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2410.15608},
}

The fine-tuned weights in this repository are released under the same MIT licence. Thanks to Mozilla Common Voice and its Welsh contributors, the Lleisiau Arfor project, Google (FLEURS) and Bangor University's Language Technologies Unit for the data.