VeuCat - Catalan and Spanish ASR, trained from scratch

VeuCat is a compact encoder–decoder automatic speech recognition (ASR) model for Catalan (ca) and Spanish (es). It uses a Whisper Base-style architecture, but both the model weights and the text tokenizer were trained from scratch using Mozilla Common Voice Scripted Speech 27.0. It was not fine-tuned from an OpenAI Whisper checkpoint.

This release represents 500,000 optimizer updates, with three consecutive training phases and an unchanged 80% Catalan / 20% Spanish training-sampling ratio. The objective is speech-to-text transcription without timestamps, not speech translation or speech synthesis.

Key results

All reported WER/CER values are percentages; lower is better. RTF is the inference-time-to-audio-duration ratio measured in the evaluation setup (lower is faster).

Mozilla Common Voice 27.0 — Catalan

Mozilla Common Voice 27.0 — Spanish

Additional Catalan evaluation datasets

The following datasets were not used as training sources. Their results are useful as an indication of domain transfer beyond Common Voice; they are not directly comparable to Common Voice because the recording conditions and text distributions differ.

The number of evaluated recordings matches the published Projecte Aina variants above. The external evaluations are reported as supplied; their preprocessing and normalization should be kept consistent when comparing other models.

Model architecture

The implementation is based on transformers.WhisperForConditionalGeneration, instantiated with a new WhisperConfig and randomly initialized weights.

Tokenizer compatibility: The custom vocabulary reserves the canonical Whisper sequence of 99 language token IDs, task/control tokens, and 1,501 timestamp tokens. These are retained for tokenizer/converter compatibility; only Catalan and Spanish were trained. Reserved language tokens do not imply multilingual recognition in the other languages, and reserved timestamp tokens were not trained to produce reliable timestamps.

Unlike standard OpenAI Whisper checkpoints, this model has a different vocabulary and token-ID mapping. Always use the tokenizer distributed with this model; do not substitute a tokenizer from openai/whisper-*.

Training data and preparation

The only training speech dataset was Mozilla Common Voice Scripted Speech 27.0 (corpus release cv-corpus-27.0-2026-09-11), using its Catalan (ca) and Spanish (es) train.tsv partitions.

Data preparation followed the project's prepare.py and train.py scripts:

1. Read the official language-specific train.tsv, dev.tsv, and test.tsv manifests and resolve audio clip paths.
2. Normalize transcript text to Unicode NFC and normalize whitespace; reject invalid, missing, or empty examples.
3. Prioritize evaluation partitions when removing duplicate audio paths across the splits within a language.
4. Decode audio with FFmpeg, convert to mono 16 kHz floating-point PCM, and cap each training example at 30 seconds.
5. Extract the standard 80-bin Whisper log-Mel features.
6. Train the ByteLevel BPE tokenizer from scratch, using reservoir samples of up to 250,000 training sentences per language (vocab_size=24000, min_frequency=5, max_token_length=24). Add Whisper-compatible special tokens after the text vocabulary.

The training manifest's dataset wrapper forms groups of five logical samples: four Catalan and one Spanish. This gives an exact 80:20 presentation ratio per five samples. It may repeat samples when one language's underlying manifest is exhausted; 80:20 is a sampling ratio, not a statement about the raw number of files or hours per language.

The training target for each clip is:

<|startoftranscript|> <|ca|> or <|es|> <|transcribe|> <|notimestamps|> TEXT <|endoftext|>

The decoder predicts the language marker and transcription task tokens as part of the sequence. Transcript text is limited to 160 BPE tokens per training example. No timestamp alignment supervision or translation supervision was used.

Training procedure

Training was performed in three successive phases using Hugging Face Trainer. A new continuation phase loads the preceding phase's learned model weights, then starts a fresh AdamW optimizer and cosine learning-rate scheduler. If a phase is interrupted, its checkpoint is resumed with the saved optimizer and scheduler state instead of restarting them.

Training parameters shared across the phases:

The project used an NVIDIA RTX 5090 Laptop GPU (24 GB VRAM) for training, with a CUDA-enabled PyTorch environment. Exact throughput and training duration depend on hardware and data I/O.

Validation during training: Teacher-forced eval_loss was monitored on a fixed random sample of 256 Common Voice dev clips per language (512 total). This small internal validation set is different from the full-set WER/CER evaluations reported above. The final v4 checkpoint continued to improve Catalan WER despite a nearly flat/slightly higher teacher-forced validation loss.

Evaluation methodology

• Common Voice: evaluated against the full dev.tsv and test.tsv partitions for each language of v27.0, rather than the 512-clip training-time validation subset.
• FestCat and OpenSLR69: separately evaluated Catalan datasets used to assess out-of-domain performance. Their published Hugging Face dataset splits are labeled train; that label refers to the dataset packaging, not their use in training this model.
• Language: supplied explicitly as ca or es, with transcription (transcribe) and timestamps disabled.
• Decoding and inference: the reported Common Voice evaluator used batched Transformers generation with beam size 1 (greedy decoding), CUDA/FP16, and a fixed maximum decoder length.
• WER/CER: computed with jiwer after Unicode NFC normalization, lowercasing, normalizing apostrophe variants and Catalan middle-dot spelling, removing most punctuation, and collapsing whitespace. Apostrophes, hyphens, accents, and middle dots are preserved where relevant.
• RTF: measured from the evaluation implementation and batching configuration; it is not a hardware-independent benchmark.

The numbers above describe the 500K-step checkpoint, not necessarily other checkpoints or quantized variants.

Usage with Transformers

Use the matching custom tokenizer and an 80-Mel feature extractor. The example below is for a short clip of up to 30 seconds; longer audio needs an explicit segmentation/long-form strategy, which has not been validated for this no-timestamp training setup.

pip install torch 'transformers>=4.50,<5' librosa

import torch
import librosa
from transformers import (
    PreTrainedTokenizerFast,
    WhisperFeatureExtractor,
    WhisperForConditionalGeneration,
)

MODEL_ID = "YOUR_USERNAME/YOUR_MODEL_REPO"  # Replace with this model's Hub ID
LANGUAGE = "ca"  # Or "es"
AUDIO_FILE = "example.wav"

assert LANGUAGE in {"ca", "es"}
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = PreTrainedTokenizerFast.from_pretrained(MODEL_ID)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_ID)
model = model.to(device=device, dtype=dtype).eval()

# Use 80 Mel bins: do not load the frontend from Whisper large-v3 (128 Mel).
extractor = WhisperFeatureExtractor(
    feature_size=80,
    sampling_rate=16000,
    chunk_length=30,
    n_fft=400,
    hop_length=160,
)

# Explicit CA/ES language selection and no timestamps.
model.generation_config.is_multilingual = True
model.generation_config.lang_to_id = {
    f"<|{lang}|>": tokenizer.convert_tokens_to_ids(f"<|{lang}|>")
    for lang in ("ca", "es")
}
model.generation_config.task_to_id = {
    "transcribe": tokenizer.convert_tokens_to_ids("<|transcribe|>"),
    "translate": tokenizer.convert_tokens_to_ids("<|translate|>"),
}
model.generation_config.no_timestamps_token_id = tokenizer.convert_tokens_to_ids("<|notimestamps|>")
model.generation_config.forced_decoder_ids = None

audio, _ = librosa.load(AUDIO_FILE, sr=16000, mono=True)
if len(audio) > 30 * 16000:
    raise ValueError("This example supports a maximum of 30 seconds of audio")
input_features = extractor(
    audio, sampling_rate=16000, return_tensors="pt"
).input_features.to(device=device, dtype=dtype)

with torch.inference_mode():
    output_ids = model.generate(
        input_features=input_features,
        language=LANGUAGE,
        task="transcribe",
        return_timestamps=False,
        max_new_tokens=160,
        do_sample=False,
        use_model_defaults=False,  # Transformers 4.50+: avoid duplicate suppression processors
    )

text = tokenizer.batch_decode(
    output_ids, skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(text.strip())

This example explicitly uses PreTrainedTokenizerFast because the originally exported model may identify its tokenizer in tokenizer_config.json as the generic PreTrainedTokenizer, which can confuse AutoTokenizer. It also constructs the correct 80-Mel extractor rather than depending on the presence of preprocessor_config.json.

Recommended repository files

For a cleaner model distribution, upload the checkpoint weights and their matching tokenizer together with:

config.json
model.safetensors               # or the actual sharded weight files

generation_config.json
tokenizer.json
tokenizer_config.json
special_tokens_map.json
preprocessor_config.json        # recommended; missing in some v4 exports
README.md

If your model export lacks preprocessor_config.json, generate it without changing the model's weights:

from transformers import WhisperFeatureExtractor

WhisperFeatureExtractor(
    feature_size=80,
    sampling_rate=16000,
    chunk_length=30,
    n_fft=400,
    hop_length=160,
).save_pretrained("v4")  # Replace with the local model-export directory

If the tokenizer metadata still declares PreTrainedTokenizer, verify and correct the tokenizer class metadata for Transformers compatibility without retraining or replacing the vocabulary. Validate token IDs and sample outputs before changing published files.

Some Transformers 4.x versions may report a duplicate SuppressTokensLogitsProcessor warning when Whisper's custom suppression configuration is inherited again by the generic generator. This is a generation-configuration warning, not evidence of damaged weights. Do not indiscriminately clear suppress_tokens: this model suppresses unused language/task tokens by design. The example above uses the applicable Transformers compatibility flag.

Intended use and limitations

Intended: Catalan and Spanish transcription for audio up to 30 seconds per chunk, with the language provided explicitly. It can be used as a research checkpoint or as a starting point for domain-specific ASR adaptation.

Not established / not supported by training:

• Reliable word- or segment-level timestamps (the timestamp tokens exist but were not trained).
• Speech translation, speech synthesis, or recognition of the other reserved Whisper languages.
• Robust automatic language identification, multilingual code-switching, or no-speech/VAD behavior without separate validation.
• Reliable transcription of noisy conversations, overlapping voices, long recordings, or specialized terminology without task-specific tests.

Read-speech benchmarks should not be treated as guarantees for production performance. In particular, FestCat WER is substantially higher than Common Voice Catalan WER; real-world errors can vary considerably by speaker, microphone, vocabulary, and recording conditions. Transcriptions can contain omissions or substitutions, so human review is appropriate for consequential uses.

Data attribution and licensing

Training data: Mozilla Common Voice Scripted Speech 27.0 (Catalan and Spanish), distributed under CC0-1.0 via the Mozilla Data Collective. Respect the dataset's distribution terms and speaker privacy restrictions. The separate external evaluation sets have their own licenses; see their linked dataset cards (the Projecte Aina OpenSLR69 derivative is CC BY-SA 4.0).

References

• Mozilla Common Voice datasets and Common Voice project
• FestCat — Projecte Aina
• Catalan OpenSLR69 — Projecte Aina
• Original OpenSLR69 dataset
• Hugging Face model cards documentation