Tamil-Whisper: Parameter-Efficient Fine-Tuning for Tamil Speech Recognition
This repository contains the LoRA adapter weights (and merged checkpoints) for openai/whisper-small adapted for Tamil Automatic Speech Recognition (ta).
Model Details
- Base Architecture: OpenAI Whisper Small (244M parameters, 12 encoder layers, 12 decoder layers)
- Adaptation Technique: Low-Rank Adaptation (LoRA) via Hugging Face PEFT
- LoRA Parameters:
- Rank ($r$): 32
- Alpha ($lpha$): 64
- Target Modules:
q_proj,v_proj - Trainable Parameters: ~1.8% of total parameters
- Acoustic Front-End: 80-channel log-mel spectrogram (30-second windows)
- Normalizer: Custom grapheme-preserving Tamil text normalizer (preserving dependent vowel signs, pulli virama
், and code-switched Latin terms).
Evaluation Results
Exact evaluation over the complete Google FLEURS Tamil held-out test split (591 utterances, 2.13 hours of speech):
| Model | Checkpoint | Dataset Split | WER (%) | CER (%) | Repetition Rate |
|---|---|---|---|---|---|
| Whisper-small Zero-shot | openai/whisper-small |
FLEURS test (ta_in, 591 samples) |
73.85% | 25.90% | 3.05% |
| Whisper-small + LoRA ($r=32$) | best_adapter (2000 steps) |
FLEURS validation (ta_in, 374 samples) |
47.22% | 13.73% | 0.00% |
| Whisper-small + LoRA ($r=32$) | whisper-small-tamil-merged |
FLEURS test (ta_in, 591 samples) |
46.34% | 13.84% | 0.51% |
Relative Improvement on FLEURS Test Split
- Relative WER Reduction: 37.25% (from 73.85% down to 46.34%)
- Relative CER Reduction: 46.56% (from 25.90% down to 13.84%)
- Repetition Rate Reduction: 83.28% (from 3.05% down to 0.51%)
- Bootstrap 95% Confidence Intervals:
- Baseline WER: 75.70% (95% CI: [72.41%, 79.84%]), CER: 26.34% (95% CI: [24.10%, 29.14%])
- LoRA Fine-tuned WER: 47.64% (95% CI: [45.56%, 50.00%]), CER: 14.26% (95% CI: [12.76%, 16.11%])
Quickstart & Usage
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
base_model_id = "openai/whisper-small"
adapter_id = "skpranav22/tamil-whisper-small-lora"
processor = WhisperProcessor.from_pretrained(base_model_id)
base_model = WhisperForConditionalGeneration.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
# Forced decoder configuration for Tamil
forced_decoder_ids = processor.get_decoder_prompt_ids(language="tamil", task="transcribe")
Ethical Considerations & Limitations
- Linguistic Morphology: Agglutinative morphology of Tamil produces higher word-level error metrics (WER ~46%) than character-level metrics (CER ~14%).
- Phonetic Ambiguity: Fine-grained retroflex vs. dental/alveolar distinctions (ல/ள/ழ, ர/ற, ந/ன/ண) remain frequent error modes that parameter-efficient adaptation does not eliminate.
- Dataset Scope: The model was trained and evaluated exclusively on Google FLEURS (
ta_in) without dynamic data augmentation; performance on informal conversational speech, high acoustic noise, or unrepresented regional dialects may degrade. - Speaker Partitioning: Google FLEURS does not publish speaker IDs in its metadata; while text leakage was verified to be zero, speaker-disjoint splitting could not be guaranteed.