Whisper Small - Hebrew Fine-tuned
Fine-tuned openai/whisper-small for Hebrew automatic speech recognition (ASR).
Model Details
- Base model: openai/whisper-small (244M params)
- Language: Hebrew
- Training data: ~50 hours of Hebrew speech (crowd-transcribe-v5 dataset)
- Training: 3 epochs, decoder-only fine-tuning (encoder frozen), bf16
- Final train loss: 0.29
- Final eval loss: 0.418
Training Configuration
| Parameter | Value |
|---|---|
| Batch size | 8 |
| Gradient accumulation | 4 |
| Effective batch size | 32 |
| Learning rate | 1e-5 |
| Scheduler | Linear |
| Warmup steps | 500 |
| Epochs | 3 |
| Precision | bf16 |
| Trainable params | 154M (decoder only) |
Usage
With Transformers
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch
processor = WhisperProcessor.from_pretrained("dm15/whisper-small-hebrew")
model = WhisperForConditionalGeneration.from_pretrained("dm15/whisper-small-hebrew")
# Transcribe
input_features = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features, language="he", task="transcribe")
text = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
With faster-whisper (recommended for production)
Convert to CTranslate2 first:
ct2-whisper-converter --model dm15/whisper-small-hebrew --output_dir whisper-hebrew-ct2 --quantization float16
Then use faster-whisper for 4-5x faster inference:
from faster_whisper import WhisperModel
model = WhisperModel("whisper-hebrew-ct2", device="cuda", compute_type="float16")
segments, info = model.transcribe("audio.wav", language="he", beam_size=1)
text = " ".join(s.text for s in segments)
Performance
- GPU inference (faster-whisper, RTX 3090): ~100-200ms per 3-5s utterance (RTF 0.03-0.10)
- CPU inference (faster-whisper, INT8): ~300-500ms per 3-5s utterance
- Trained on 16kHz audio. For telephony (8kHz), a separate fine-tuned version is recommended.