Armenian Whisper Small (whisper-small-hy)

Fine-tuned openai/whisper-small for Eastern Armenian (hy) automatic speech recognition (ASR). Part of the Armenian speech research stack alongside HyVoxPopuli and SpeechT5 TTS.

Evaluation (from training run)

Metrics below come from the same validation split used during fine-tuning (auto-generated Trainer card). They are not a fresh benchmark on HyVoxPopuli or Common Voice test sets.

Interpretation: Validation WER remains ~75% with rising val loss at the end of training — strong signs of overfitting on a small fine-tune set. Treat this checkpoint as an experimental baseline, not production ASR.

For stronger Armenian ASR today, prefer facebook/mms-1b-all with the hy language adapter (see HyVoxPopuli dataset card).

Quick start

import torch
from datasets import load_dataset, Audio
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "Edmon02/whisper-small-hy"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to(device)
model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(
    language="hy", task="transcribe"
)

# Example: HyVoxPopuli test clip (filter empty text in real eval)
ds = load_dataset("Edmon02/hyvoxpopuli", split="test")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
ds = ds.filter(lambda x: bool((x["normalized_text"] or "").strip()))
sample = ds[0]

inputs = processor(
    sample["audio"]["array"],
    sampling_rate=16_000,
    return_tensors="pt",
)
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.no_grad():
    ids = model.generate(**inputs, max_new_tokens=256)

hypothesis = processor.batch_decode(ids, skip_special_tokens=True)[0]
print("Hypothesis:", hypothesis)
print("Reference:", sample["normalized_text"])

Pipeline (high level)

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="Edmon02/whisper-small-hy",
    device=0,
    generate_kwargs={"language": "hy", "task": "transcribe"},
)
# asr(audio_path_or_array)

Intended uses

• Prototyping Armenian ASR in Whisper-based pipelines
• Comparing Whisper fine-tuning vs MMS / XLS-R on HyVoxPopuli
• Teaching / reproducing low-resource Whisper fine-tunes

Out of scope

• Production transcription without re-benchmarking on your domain
• Real-time streaming (use faster-whisper + optimized checkpoints after validation)
• Non-Armenian languages (use base openai/whisper-small or other adapters)
• Speaker diarization or timestamp-heavy workflows without extra tooling

Training procedure

Training data is not serialized in this repo; it was likely Common Voice hy-AM (and/or project-internal splits) from the same era as speecht5_finetuned_hy. Re-run training with documented splits before trusting metrics.

Limitations

• High WER (~75%) on the recorded validation pass — not competitive with modern MMS adapters
• Small fine-tuning corpus typical of early experiments
• Literary / read speech in HyVoxPopuli differs from conversational CV audio — domain shift
• No word-level timestamps in default generate config (return_timestamps: false)
• Multilingual Whisper vocabulary; Armenian uses language token <|hy|>

Related assets

Bias and ethics

ASR errors disproportionately affect dialectal or code-switched Armenian (e.g. Russian lines in literary sources). Do not use outputs for high-stakes decisions without human review.

Citation

@misc{whisper_small_hy2024,
  author = {Avetisyan, Edmon},
  title = {Armenian Whisper Small (whisper-small-hy)},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Edmon02/whisper-small-hy}}
}

Also cite OpenAI Whisper and your training data (e.g. Common Voice, HyVoxPopuli).

License

Apache-2.0 (inherits from Whisper fine-tuning convention). Base openai/whisper-small license applies to architecture and tokenizer.