LES-eg2-665M-asr-en

English speech recognition from a frozen EmbeddingGemma 2: an 88M reader reads its audio as words of its own text-embedding space (LibriSpeech test-clean WER 1.74 greedy, 1.58 with the n-gram LM and reranker; Results).

EmbeddingGemma 2 ("eg2" in the model name) is used exactly as it is: its weights are not changed and it gives the same embeddings as before. A trained reader of 88M parameters turns the hidden states of its audio encoder and backbone into word-level CTC outputs, scored against EmbeddingGemma 2's own text embeddings of 407,736 words.

• Accuracy: LibriSpeech test-clean WER 1.74 greedy, 1.61 with the n-gram LM and 1.58 with the reranker (Open ASR Leaderboard normaliser); seven more English test sets in the Results table.
• Vocabulary control: add words, or replace the whole word table, without retraining (Your own word table).
• Raw outputs for your own re-ranking: the scored 32-best list, every frame's word candidates, scoring of any hypothesis and a search over your own candidates (Re-ranking).
• Embeddings from the same pass: EmbeddingGemma 2's own audio and text embeddings (EmbeddingGemma 2 embeddings).
• Permissive: trained only on permissively licensed data (no GigaSpeech, SPGISpeech, TED-LIUM or Fisher), so the model files can be used commercially under Apache License 2.0 (Training data, NOTICE).

How it works

EmbeddingGemma 2 runs as its audio-language configuration (577M parameters in the paper): the audio encoder (two downsampling convolution layers and twelve conformer layers, USM architecture) turns Mel filterbank inputs into 25 soft tokens per second (d_A = 1536), the audio adapter projects them to the backbone's model dimension (d_M = 512), and the 24-layer bidirectional backbone processes them. The reader reads, for every soft token, the backbone input and its 24 layer outputs, the twelve conformer layers and the audio encoder output; it gives a blank logit and a unit vector that is compared by cosine with the word table, i.e. EmbeddingGemma 2 text embeddings (mean pooling, linear projection to d = 768, l2-normalisation) of every word.

Speech and words therefore meet in one embedding space. Each 40 ms soft token becomes a vector z(t) among the text embeddings of the vocabulary; greedy decoding emits the nearest word row when s · cos beats the blank logit, and the beam decoders keep the 16 nearest rows of every frame as candidates. The figure shows real cosines for one frame of "carriage" in a LibriSpeech dev-clean utterance, zoomed in twice: from the whole space to the target around z(t), then to the scores. The nearest row is "carriage" (cosine 0.46); the next ones, at 0.31-0.32, are words related in meaning (chariot, hearse) and spelling variants found in the vocabulary (carrage, carridge), all far below "carriage" and close to the blank head's score b(t)/s = 0.31.

The word table (word_table.safetensors, 407,736 x 768, stored as row-wise INT8) is a cache of EmbeddingGemma 2 text embeddings, so it is not counted again. The vision encoder (167M) is not loaded.

Usage

pip install "torch>=2.11" torchvision "transformers>=5.20" safetensors numpy soundfile soxr
# torchvision: EmbeddingGemma 2's processor loads its image / video parts even for audio
pip install https://github.com/kpu/kenlm/archive/master.zip     # only for the beam decoders with the n-gram LM
pip install "sentence-transformers>=6.1"                         # only for add_words / set_vocabulary

import soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("lee1jun/LES-eg2-665M-asr-en", trust_remote_code=True).to("cuda").eval()
audio, sr = sf.read("speech.flac", dtype="float32")

model.transcribe(audio, sampling_rate=sr)                    # greedy: "the transcript"
model.transcribe(audio, sampling_rate=sr, decoder="beam")    # + n-gram LM (downloads lee1jun/LES-eg2-asr-en-lm-4gram)
model.transcribe(audio, sampling_rate=sr, decoder="rerank")  # + the beam's 32-best list re-ranked by the reranker heads
model.transcribe(audio, sampling_rate=sr, decoder="rerank", profile="no_lm")  # second pass without the n-gram LM
model.transcribe([a1, a2, a3])                               # batches, any lengths

r = model.transcribe(audio, sampling_rate=sr, return_timestamps=True)
# {"text": ..., "duration": ..., "words": [{"word", "start", "end", "confidence"}, ...]}
r = model.transcribe(audio, sampling_rate=sr, decoder="rerank", return_nbest=True, return_candidates=True)
# {"text": ..., "nbest": [{"text", "words", "score", "ctc", "lmf", "n", "reranker", "chosen"}, ...],
#  "candidates": [{"time", "blank", "words": [(word, cosine, log p), ...]}, ...]}

EmbeddingGemma 2 (google/embeddinggemma-2, revision 914f7f89142e33e77833254d9c9b90c3cef7303b) is downloaded on first use and used frozen, without its vision encoder; its weights are not part of this repository. Audio at any sampling rate is resampled to 16 kHz.

Decoding options

model.second_pass(lm, decoder=...) returns the beam decoder as an object for repeated use (build it again after changing the word table).

Your own word table

model.add_words(["embeddinggemma", "huggingface"])           # new vocabulary without retraining
model.set_vocabulary(["turn", "left", "right", "stop"])       # own word table: outputs only these words
model.reset_vocabulary()                                      # back to the shipped table

The decoders can only output words of the table, so the table is also a vocabulary control. add_words extends it. set_vocabulary(words) replaces it with exactly your list (commands, names, a domain glossary). Words already in the table keep their rows; other words get rows from EmbeddingGemma 2 by the training recipe (needs sentence-transformers). The reader is not retrained, and reset_vocabulary() restores the shipped table. Speech outside the list comes out as the nearest listed words or, mostly, as nothing; there is no explicit rejection. The beam decoders still apply the n-gram LM. model.save_pretrained(dir) saves the custom table (without this card).

EmbeddingGemma 2 embeddings

The model also returns EmbeddingGemma 2's own embedding (768-d, L2-normalised: the vector of SentenceTransformer("google/embeddinggemma-2").encode(...)), alone or together with the transcript:

r = model.transcribe(audio, sampling_rate=sr, return_embedding=True)   # {"text": ..., "embedding": tensor [768]}
e = model.embed_audio(audio, sampling_rate=sr)                         # embedding only (no reader, no word table)
q = model.embed_text("quarterly revenue guidance", prompt_name="query")  # text in the same space
score = q @ e                                                           # cosine: text-to-audio search
model.embed_audio([a1, a2], dim=256)                                    # Matryoshka sizes 512 / 256 / 128

return_embedding pools the EmbeddingGemma 2 pass the decoder already runs, so for audio up to 40 s the embedding costs nothing extra (audio decoded in windows, and audio of 29-30 s, takes one more EmbeddingGemma 2 pass). Up to 30 s the vectors equal sentence-transformers' (cosine ≥ 0.997, bf16 rounding). Longer audio is mean-pooled over 25 s pieces so that the whole input counts, whereas sentence-transformers encodes only the first 30 s. embed_text takes EmbeddingGemma 2's prompt names ("query", "document", ...; default none) or a literal prompt= and needs no sentence-transformers.

Results

• Metric: word error rate (%) with the Open ASR Leaderboard text normaliser. Test sets: LibriSpeech test-clean / test-other, TED-LIUM 3 test (1,142 utterances) and the Open ASR Leaderboard test sets of VoxPopuli, Earnings-22, AMI (IHM), GigaSpeech and SPGISpeech. Mean is the mean of the eight WERs. Params: every weight loaded for inference; for this model, in parentheses, the weights trained here (the 88M reader, plus the 6.44M reranker heads for "rerank"), EmbeddingGemma 2 being frozen.
• Beam decoders: the CC BY 4.0 word 4-gram of the LM repository (alpha 0.15, beta 1, beam 64). "rerank" re-ranks the beam's 32-best list with the reranker heads and the weights in decoding.json.
• Tuning: one setting for all eight sets, chosen on LibriSpeech, VoxPopuli and AMI development audio. No test set and no Earnings-22 audio or text was used for training or tuning. Tuning each domain on its own development split did not help on average (mean 6.62 instead of 6.60 with the reranker).
• Reproducibility: computed with the research implementation on these weights (full-precision word table, 120 s batches). This repository's remote code (INT8 table) reproduces the greedy numbers on LibriSpeech: 2,582 / 2,620 and 2,844 / 2,939 transcripts identical, WER 1.74 / 4.49 against 1.74 / 4.51. "beam" and "rerank" give the research pipeline's transcript for 1,549 of 1,600 spot-checked decodes (200 utterances each of 4 test sets).
• Without the n-gram LM: profile="no_lm" (no LM download, no kenlm; "rerank" uses its own reranker heads) gives means of 6.86 ("beam") and 6.84 ("rerank"), against 6.67 and 6.60 with the LM: the n-gram LM brings most of the second pass's gain.
• Other models: greedy decoding of the public checkpoints, run by us on the same audio, references, normaliser and scorer (Parakeet: NeMo; Whisper: transformers, fp16, English, Whisper's long-form decoding for the 0.1 % of inputs over 30 s). Params counts every weight loaded for inference. They were trained on other data (see their model cards). The scorer is the current leaderboard code, which merges split compounds ("shop boy" = "shop-boy"), so every row is lower than figures computed with a plain word edit distance (Parakeet-TDT on LibriSpeech test-clean: 1.29 here, 1.66 without merging).

N-best candidates

Each 40 ms frame offers several word rows; the beam search takes the 16 nearest per frame (the figure shows the top 5 for four words of a LibriSpeech dev-other utterance). Greedy decoding takes the top row of every frame, the beam search with the n-gram LM and the reranker choose a path through the candidates, and the oracle is the best path the list contains.

The beam's 32-best list (31.5 distinct hypotheses on average) holds much better transcripts than the decoders pick. The table gives the WER if the best hypothesis among the first k were picked (oracle WER, same normaliser and scorer):

Choosing the best of the top 10 would lower the mean WER from 6.67 to 4.01. The included reranker recovers 0.07 of that gap (mean 6.60); see Your own re-ranking and search.

Re-ranking

The reranker

decoder="rerank" re-scores the beam's 32-best list: each hypothesis gets score = 1 · ctc + 0.2 · lmf + 1 · n + 2 · reranker (the weights in decoding.json), and the largest wins. ctc is the hypothesis' CTC log-likelihood over the frame candidates, lmf its n-gram log-probability (natural log, with the end of sentence), n its word count and reranker the score of the reranker heads (reranker_heads.safetensors).

• What it reads. A small Transformer (3 pre-norm layers, d = 256, 4 attention heads; 2.15M parameters per head, three heads from three training seeds whose scores are averaged, 6.44M in all) reads one hypothesis at a time. Each word is one token, the sum of its EmbeddingGemma 2 table row, the reader's frame vector at the word's best frame in its Viterbi CTC alignment (and the cosine between the two), and 14 numbers: the word's log-probability at that frame, whether and where it is among the frame's 16 candidates, the frame's blank log-probability, the frame's best log-probability and the margin to it, the length of the word's CTC segment and the gap to the previous word, its n-gram log-probability, a function-word flag, its length in characters, its position in the utterance, and two N-best consensus shares (the part of the list, weighted by the softmax of the hypotheses' scores, that has the same word within two frames, or anywhere). A summary token carries the hypothesis' ctc, lmf and n relative to the beam's best hypothesis; it gives the score.
• How it was trained. Listwise: for each utterance, the softmax of the scores over its 32-best list is trained towards softmax(−word errors / τ) with a cross-entropy loss, so the heads learn to rank hypotheses by their errors. The lists come from about 23,000 held-out utterances of AMI, VoxPopuli and The People's Speech (audio and text outside the training data of the reader and of the n-gram LM, so the lists show real errors), decoded by this model with the n-gram LM; permissive data only, no test or Earnings-22 audio. The four weights were then tuned on the pooled LibriSpeech, VoxPopuli and AMI development audio.
• Without the n-gram LM (profile="no_lm"): reranker_heads_nolm.safetensors has the same architecture and training, on 32-best lists decoded without any n-gram LM (alpha 0, beta 0), with the LM inputs (lmf and each word's n-gram log-probability) set to zero; the consensus shares weight the hypotheses without the LM, and score = ctc + 2 · reranker.
• What it brings. Mean WER 6.67 → 6.60 with the LM and 6.86 → 6.84 without.
• Caveats. The consensus shares and the scores relative to the beam's best depend on the list, so reranker scores compare only within one list. Trained on three domains; on SPGISpeech it is slightly worse than the plain beam (3.68 vs 3.56).

Your own re-ranking and search

The 32-best list holds much better transcripts than the decoders pick. Choosing the best hypothesis of the list would give a mean WER of 3.31 (best of the first 10: 4.01; AMI 10.75 → 4.20, Earnings-22 12.82 → 7.96; see the oracle table), against 6.60 with the included reranker, which recovers only 0.07 of the 2.66 points between the best beam (6.67) and the best of the first 10. Better re-ranking is where this model has the most room: a stronger language model, domain knowledge, or a better learned reranker can turn much of that gap into real gains, so please try your own. The decoders return their own scores, so you can recombine them, add your own, or run your own search:

import numpy as np

# 1. your own weights or features on the 32-best list (here an external LM score of each text, my_lm)
r = model.transcribe(audio, sampling_rate=sr, decoder="rerank", return_nbest=True)
best = max(r["nbest"], key=lambda h: h["ctc"] + 0.5 * h["lmf"] + h["n"] + 2 * h["reranker"] + my_lm(h["text"]))

# 2. hypotheses from anywhere (an LLM's correction, another recogniser), scored together with the 32-best list
s = model.score_hypotheses(audio, ["the corrected sentence"], sampling_rate=sr)
# the 32-best entries (the same scores as above) + yours: {..., "given": True, "in_nbest": False, "oov": []}

# 3. your own search over the frame candidates, without another audio pass
c = model.transcribe(audio, sampling_rate=sr, return_candidates=True, candidates_format="arrays")["candidates"]
c["logp"][c["ids"] == model.vocab.index("uh")] = -np.inf         # e.g. forbid a word, or bias / re-weight "logp"
nbest = model.beam_search(c, alpha=0.3, beta=0.5, nbest=10)      # any KenLM binary / ARPA file via lm=

• return_nbest entries (beam order) hold the table words, the parts ctc, lmf, n and reranker (None for "beam"), the decoder's score (ctc + alpha · lmf + beta · n for "beam"; the weighted sum above for "rerank") and chosen. For "rerank", recombining the parts with decoding.json's weights reproduces the decoder's choice; for "beam" the choice is the search's own first hypothesis, whose pruned search score can rank differently from score.
• return_candidates gives, per frame, time (shifted back like word times), blank (its probability) and words as (word, cosine, log p), best first; p is the softmax over the blank logit and s · cos with every row, as the beam decoders' candidates. candidates_format="arrays" gives numpy arrays over every frame: time and lp_blank [T], ids (indices into model.vocab), cos and logp [T, top_k], plus the fingerprint of the word table they index.
• score_hypotheses(audio, hypotheses, decoder="rerank", include_nbest=True, lm=, alpha=, beta=, weights=) runs one EmbeddingGemma 2 pass (inputs up to 40 s; long_form_s=None for longer ones) and scores each hypothesis, a string or a list of table words, like an N-best entry. With include_nbest it returns the beam's own 32-best entries unchanged, then each of your hypotheses scored as one more entry of that list, so yours change neither the 32-best scores nor each other; include_nbest=False scores only yours, as one list. Words outside the table are listed in oov (never a candidate, read as a zero row by the reranker: add_words them first); chosen marks the largest score.
• beam_search(candidates, lm=, alpha=, beta=, beam=, nbest=) is the beam decoders' search on candidates_format="arrays" candidates (one dict or a list); on unchanged candidates of an input up to 40 s it returns decoder="beam"'s N-best list. Candidates made before add_words / set_vocabulary are refused.
• profile="no_lm" works for score_hypotheses and beam_search too (no kenlm needed).

Training data

Segments over 29 s were dropped. Transcripts are lower-cased and normalised to words; an utterance is kept only if every word is in the LibriSpeech vocabulary or occurs at least 3 times in these corpora. Sessions and videos that overlap evaluation sets were removed (Open ASR VoxPopuli test sessions; word 10-grams shared with evaluation references, e.g. YouTube re-uploads of GigaSpeech test talks).

Training: the reader was trained on LibriSpeech 960 h, then on the mix above (2 epochs on a 27.7k h subset, then 4 epochs on all of it), with EmbeddingGemma 2 frozen throughout. A 17,000-step partial run of the mix (about half an epoch) was the starting point of the final 4 epochs. The 4 epochs follow one cosine learning-rate schedule. Greedy mean WER of the eight sets at the start of epochs 3 and 4 and at the end: 7.35, 7.01 and 6.89; the last epoch still improved every set.

Limitations

• English only; lower-case words without punctuation or casing. Numbers come out as words.
• Non-streaming. Word times are spike-based estimates of word centres.
• Words outside the table can only be produced after add_words.
• Pseudo-labelled training data (Granary, Libri-light) carries the errors of the labelling models.

Acknowledgements and related work

This project started from an idea we took from the Embedding Gemma Transcribe Mac app: reading EmbeddingGemma 2's audio as words of its own text space. We thank the authors of the projects below.

Readers that score EmbeddingGemma 2 text embeddings of words (as this model does):

Other speech recognisers on the EmbeddingGemma 2 audio encoder (subword or character outputs):

Licence

Code: Apache 2.0 (LICENSE-CODE). Model files: Apache License 2.0 (LICENSE). EmbeddingGemma 2 keeps its own terms and the Gemma Prohibited Use Policy; the n-gram LM (lee1jun/LES-eg2-asr-en-lm-4gram) is CC BY 4.0. Attributions: NOTICE.

Citation

If you use this model, please cite it:

@misc{les_eg2_665m_asr_en,
  title        = "{LES-eg2-665M-asr-en}: English Speech Recognition from a Frozen {EmbeddingGemma 2} with a Word-Embedding {CTC} Reader",
  author       = {lee1jun},
  year         = {2026},
  month        = nov,
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/lee1jun/LES-eg2-665M-asr-en}},
  note         = {n-gram LM: \url{https://huggingface.co/lee1jun/LES-eg2-asr-en-lm-4gram}}
}

This model builds on EmbeddingGemma 2:

@article{embeddinggemma2,
  title   = {EmbeddingGemma 2: Unified and Efficient Multimodal Representations},
  author  = {Schechter Vera, Henrique and Dua, Sahil and others},
  journal = {arXiv preprint arXiv:2610.37364},
  year    = {2026}
}

Files

Author

lee1jun (contact: lee1jun@kaist.ac.kr)