EmbeddingGemma 2 ASR — streaming Hebrew + English speech recognition

This model transcribes Hebrew and English, both from files (batch) and as people speak (streaming). It is built on the audio encoder of EmbeddingGemma 2 with a small streaming reader and a transducer head on top, and it needs no external language model.

• 351M parameters, about 2.3× smaller than Whisper Turbo.
• 2–5× faster than Whisper Turbo in batch on the same GPU (RTFx up to 604).
• Streaming in 0.24 s steps, and up to ~256 streams at once on one RTX 5090.
• Hebrew and English in one model.

Word Error Rate

Whisper Turbo = ivrit-ai/whisper-large-v3-turbo, the strongest open Hebrew model we know of. It cannot stream: it needs whole 30-second windows. Both models are scored with the ivrit.ai leaderboard text normalisation.

RTFx

RTFx is seconds of audio transcribed per second of compute (higher is better), measured as on the Open ASR Leaderboard: audio decoding is not timed, everything after it is. Both models ran on the same RTX 5090, each at its fastest setting.

Real-time streams on one GPU

Many streams run at once on one GPU. Each round, every stream gets its next piece of audio, and the round has to finish before the next piece arrives. Below the dashed line, every stream keeps up.

Accurate streaming re-reads the last 25 seconds with full context every 0.24 s, and a word becomes final once two passes agree. Fast streaming reads each 0.48 s chunk once.

Quick start

pip install -U huggingface_hub
hf download MaorB/embeddinggemma-2-asr-hebrew-english-streaming --local-dir embeddinggemma-2-asr-hebrew-english-streaming && cd embeddinggemma-2-asr-hebrew-english-streaming
pip install -r requirements.txt

from kolstream import KolStream

asr = KolStream.from_pretrained(".")          # also downloads google/embeddinggemma-2 (the base encoder)
print(asr.transcribe("speech.wav"))           # batch: any length, any sample rate

Streaming, for example from a microphone sending 48 kHz audio in 0.24 s pieces:

import numpy as np

stream = asr.stream(sample_rate=48000)        # mode="accurate" (default) or mode="fast"
for piece in pieces:                          # float32 numpy arrays
    out = stream.push(piece)
    print(out["final"], "|", out["interim"])  # interim = words that may still change
print(stream.push(np.zeros(0, np.float32), final=True)["final"])

examples/transcribe.py and examples/live_file.py are runnable versions. A GPU is recommended; the model also runs on CPU, slower.

How it works

1. Encoder. EmbeddingGemma 2's audio tower (a causal conformer, 40 ms frames) turns audio into frames. Its top two layers and output projection were fine-tuned for speech recognition; the rest are the original weights, downloaded from Google's repository at load time.
2. Reader. A 43M-parameter transformer reads the encoder frames. It was trained both with full context and with chunked attention, so one set of weights serves batch and streaming.
3. Transducer. A small transducer head (stateless predictor) writes word pieces from a 2,048-piece Hebrew + English vocabulary.

Streaming runs the encoder incrementally with cached layer states, so audio is never encoded twice.

Training data

Drawn from about 19,000 hours of transcribed speech:

• Hebrew podcasts from ivrit.ai audio-v2, labelled by ivrit.ai's Whisper model (transcripts). The podcast used in eval-d1 is excluded.
• Human-transcribed Hebrew from crowd-transcribe-v5 and crowd-recital.
• About 1,300 hours of Knesset plenums.
• English from LibriSpeech (960 h).

In the last training stage, a quarter of the audio was made to sound like phone voice notes (band-limiting, compression, noise, reverb). Thanks to ivrit.ai for the Hebrew data; it is used under the ivrit.ai license, which permits training AI models.

Limitations

• No punctuation or capital letters; numbers can come out as digits or words.
• Less accurate than Whisper Turbo: about 1 point on podcasts, 3 points on voice notes. Formal vocabulary and foreign names (as in FLEURS) are the weakest area.
• English was trained on read audiobooks, so conversational English is weaker than LibriSpeech suggests.
• Most Hebrew training labels come from a Whisper model, so the model inherits some of its habits.
• Do not use it to identify or impersonate speakers. The ivrit.ai license forbids using its data for voice deep-fakes.

Reproducing the numbers

benchmarks/bench.py measures WER, RTFx and streaming capacity. It includes the batched multi-stream loop used for the capacity numbers. benchmarks/bench_whisper.py runs the Whisper Turbo comparison. Test sets: eval-d1 (1.43 h), eval-whatsapp (1.18 h), FLEURS he_il test (2.05 h), LibriSpeech test-clean (5.4 h).

The WER table uses the most accurate decoding. RTFx and stream counts use the fastest, which scores about 0.3 points higher WER: podcast 6.1%, WhatsApp 10.4%, FLEURS 24.3%, LibriSpeech 4.1%.

License

Apache 2.0, the same as the EmbeddingGemma 2 base model. The training data is from ivrit.ai (ivrit.ai license: CC BY 4.0 with an AI-training purpose clause) and LibriSpeech (CC BY 4.0). The text normaliser in benchmarks/hebnorm.py is from ivrit.ai's asr-training repository (MIT).