whisper-small-ha-en-direct-pilot

Experimental pilot checkpoint — not production quality. A LoRA adapter over openai/whisper-small (task=translate mode), fine-tuned directly on Hausa audio -> English text pairs from McGill-NLP/NaijaS2ST, skipping the Hausa-text intermediate step entirely (unlike the project's main cascade at nahomazmach/whisper-small-ha

• NLLB-200).

Why this exists

The project's cascade (Hausa ASR -> NLLB-200 translation) showed that translation quality collapses when NLLB receives real ASR output instead of clean Hausa text (BLEU ~22-25 on gold text vs ~8-10 on real ASR output) — the bottleneck is ASR error propagation, not the translation model. This pilot tests whether skipping the written-Hausa intermediate step avoids that problem.

Training details

• Data: 256 train / 128 validation examples (a targeted 2-shard subset of NaijaS2ST's train split, selected via metadata-only audit to avoid an unnecessary ~69GB full-split download — see the source repo's direct_pilot/RESULTS.md for the full methodology).
• Method: LoRA (rank 16), 1.77M trainable parameters (0.7% of the 243M total), 50 steps / 3.125 epochs.

Results

Far below the cascade's real-ASR BLEU of ~8-10. Given the tiny training set, this most plausibly reflects insufficient training data rather than a fundamental weakness of the direct paradigm — see the source repository for full discussion.

Not evaluated on the official NaijaS2ST dev split (validation here is derived from the train split) — treat this as a feasibility pilot, not a final benchmark number.

Usage

import torch
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

processor = WhisperProcessor.from_pretrained("nahomazmach/whisper-small-ha-en-direct-pilot")
base_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(base_model, "nahomazmach/whisper-small-ha-en-direct-pilot")

# input_features = processor.feature_extractor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features, language="hausa", task="translate", max_new_tokens=225)
english = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)[0]