EmbeddingGemma 2 — Persian speech ↔ text
google/embeddinggemma-2 fine-tuned so that a Persian speech clip and its transcription land next to each other in the embedding space.
Use it to search Persian audio with text, match a spoken utterance to the closest sentence or intent in a list you provide, or find duplicate/similar audio. It is not a speech-to-text model. It ranks candidate texts you provide and cannot write out sentences it hasn't been given. For transcription use a generative ASR model (e.g. Whisper).
Results
Task: given a clip, rank every distinct sentence in the test set; the metric is the share of clips whose own transcription ranks first (R@1). Sentences are deduplicated after Persian normalization (ي/ی, ك/ک, ZWNJ, diacritics, punctuation).
| Test set | Clips / candidate sentences | Base R@1 | Fine-tuned R@1 | Fine-tuned R@5 | Text→audio R@1 (base → ft) |
|---|---|---|---|---|---|
| Common Voice 17 fa test (speakers unseen in training) | 10,558 / 10,457 | 64.0% | 98.5% | 99.3% | 74.1% → 98.0% |
| YouTube, 82 videos fully held out | 1,326 / 1,326 | 93.9% | 99.5% | 99.7% | 93.6% → 98.9% |
| Filimo test (5k sample, movie dialogue, ~2 s clips) | 5,000 / 4,856 | 34.4% | 90.7% | 94.9% | 46.8% → 89.7% |
| FLEURS fa test | 871 / 324 | 99.2% | 100% | 100% | 100% → 100% |
On clips whose sentence never occurs in the training data: Common Voice 98.5%, YouTube 99.5%, filimo 92.8% (base: 64.2 / 93.9 / 34.7). About 28% of filimo test sentences also appear in filimo train. Those are mostly short stock phrases such as «بله» or «سلام», and they score lower than novel sentences, so the overlap isn't inflating the filimo number.
The model still works outside Persian. Zero-shot English Speech Commands (35 words) is 91.1%, vs 93.0% for the base model.
Usage
import torch
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("MohammadGholizadeh/embeddinggemma-2-persian-speech",
model_kwargs={"torch_dtype": torch.bfloat16}, # bf16 or fp32, never fp16
config_kwargs={"vision_config": None}) # audio + text only
audio = model.encode(["clip1.wav", "clip2.wav"]) # paths, or {"array": np.ndarray, "sampling_rate": 16000}
texts = model.encode(["سلام، حالت چطوره؟", "قیمت این محصول چنده؟"], prompt_name="SearchQuery")
print(model.similarity(audio, texts))
- Audio: mono, 16 kHz, no prompt. Trained on 0.5–20 s clips.
- Text: use
prompt_name="SearchQuery", the prompt used in training. Normalizing ي→ی and ك→ک in your texts helps.
Training
- Data: 479k (audio, transcription) pairs, ~534 h. Filimo train (322k), YouTube (128k, held-out videos excluded), Common Voice 17 fa train (29k). All resampled to 16 kHz.
- Loss:
CachedMultipleNegativesRankingLoss(audio anchor → transcription positive, in-batch negatives), batch 256 with no duplicate texts per batch, mini-batch 32. - Optimization: 1 epoch (1,872 steps), lr 2e-5 cosine with 5% warmup, bf16 autocast over fp32 weights. About 3 h on one GH200.
- Validation R@1 (1k-sentence pools) plateaued after ~1,000 steps at 99.8% CV / 99.6% YouTube / 93.2% filimo.
Limitations
- Retrieval and matching only; no transcription.
- Short, generic utterances (“بله”, “نه”) are inherently ambiguous across speakers and recordings.
- Evaluated on read speech, YouTube talks and movie dialogue. Noisy phone-call audio and strong dialects weren't tested.
- General text–text retrieval quality after fine-tuning wasn't measured.