EmbeddingGemma 2 Transcribe — reader v2

A 19.6 M-parameter reader that turns the audio soft tokens of Google's EmbeddingGemma 2 into English words: each 40 ms token is mapped into the model's 768-d text-embedding space and read as the closest of 200,000 words, or as blank, with greedy CTC decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained. Reader v2 is trained on the three LibriSpeech training sets (960 h): test-clean WER 4.21 %, test-other 10.42 %.

The idea comes from Shahir's post (2026-10-13) and the Embedding Gemma Transcribe Mac app (eg2.haku.sh), whose code and weights are not public. This reader is an independent reproduction, trained from scratch on LibriSpeech. It is not affiliated with Google.

Reader v1 (4 layers, 13.3 M parameters, trained on train-clean-100 alone; test-clean 6.81 %, test-other 15.10 %) is revision 1af9c5f5 of this repository.

These weights are framework-neutral (a PyTorch state dict and a NumPy table). The converted builds that run them on a device:

Files

SHA256SUMS lists every file. RELEASE.md is the release note of the reader/ directory of the eg2-transcribe repository; the paths in it (train/, fixtures/) are relative to that directory.

Contract

• Input: EmbeddingGemma 2 audio soft tokens, float32 [T, 512], 25 per second: the embed_audio output of google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b (model.get_audio_features(input_features, input_features_mask).pooler_output; transformers 5.19.0 with the default attention, not eager, whose audio mask is inverted in that version).
• Body: LayerNorm → Linear 512→512, 6 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU, rotary positions with base 10000), LayerNorm → Linear 512→768. Attention covers the whole input window in both directions. 19,573,762 parameters.
• Output per token: a unit vector in the 768-d text space and a blank logit.
• Logits: word w scores scale × cos(vector, word_table[w]), with scale = exp(log_scale) = 216.351 (learned); blank is class 200,000 and scores blank_logit.
• Decoding: per token, blank where blank_logit > top-1 word logit (a tie reads the word), else the top-1 word; merge repeats, drop blanks. No language model, no punctuation, no casing.
• Long audio: 30 s encoder windows (750 tokens), each starting 96 tokens (3.84 s) before the end of the previous one, keeping only its new tokens and dropping its last token: the encoder is causal, so the start of a window lacks its left context and the last token of a 30 s cut misses one input frame. reader_config.json → window.
• Weights file: reader.pt keys body.* (the reader body), blank.weight [1, 768], blank.bias [1], log_scale; the module is Reader(arch, n_vocab, k, layers) of reader/train/model.py in the eg2-transcribe repository (arch a, layers 6; reader_config.json → architecture.layers).

WER

Greedy, no language model, every utterance of each set. Reference and hypothesis are lower-cased with runs of spaces collapsed; words outside the vocabulary stay in the references and count as errors.

Long audio, in the 30 s windows of the contract: The Adventure of the Speckled Band (LibriVox, 3,257.4 s, 125 windows), scored against the Project Gutenberg text (9,889 words; the LibriVox intro and outro are in the audio only): 3.81 % through the float32 PyTorch path, 3.80 % through the Core AI build (the encoder on the GPU and this reader in float16 on the Neural Engine of an Apple M4 Max, two windows in flight). Reader v1: 5.78 % and 5.80 %.

Training

• Data: LibriSpeech train-clean-100, train-clean-360 and train-other-500 (CC BY 4.0), 281,241 utterances, 961.6 h, read as one set per epoch. The 19,755 utterances with a word outside the vocabulary were left out: 261,486 utterances, 890.2 h, 80,121,094 tokens. Soft tokens computed once in float32 (TF32 off), stored as float16.
• dev-clean chose the reader body and the epoch; test-clean and test-other were decoded once, after training.
• Objective: CTC over the 200,000 words and blank (torch.nn.functional.ctc_loss, float32, zero_infinity, mean of loss / target length).
• Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, 500 warmup steps, then cosine to zero at step 163,430; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 weights and activations, the training matmuls in TF32 (A100); the dev, test and fixture WER are decoded in full float32.
• 9 epochs, 147,087 steps; the best dev-clean WER came at epoch 7 (these weights); early stop after 2 epochs without a better dev-clean WER.
• Hardware: one NVIDIA A100-SXM4-80GB (Google Colab), torch 2.11.0+cu130, CUDA 13.0; 4.38 h of training plus 2 min of dev decoding.
• Body chosen on dev-clean between two bodies trained the same way: 4 layers (13.3 M parameters, 3.88 % at its best epoch 8) and 6 layers (19.6 M, 3.75 % at epoch 7, these weights).

The command of the released run is in RELEASE.md.

Word table

• vocab.txt: every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by code point, apostrophes kept.
• word_table.npy: the EmbeddingGemma 2 text embedding of each word with the prompt task: search result | query: (sentence-transformers, float32, then stored as float16 and L2-normalized). Each of 100 random words, embedded again, comes back at rank 1. Both files are those of v1.
• Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %). These are always errors.

Limits

• By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 1.8 % for words of 1–2 tokens (up to 80 ms), 2.0 % for 3–5 tokens and 6.1 % for 6 or more.
• The most frequent substitutions on test-clean are function words (and → in 27, in → and 22, a → the 16); homophones are 12.9 % of all substitutions, which a reader without a language model cannot tell apart.
• English read speech only (audiobooks); trained on utterances of up to 29.7 s and measured on utterances of up to 35.2 s; longer audio goes through the 30 s windows above.

License and credits

• This repository: Apache-2.0 (LICENSE); see NOTICE.
• LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on train-clean-100, train-clean-360 and train-other-500.
• LibriSpeech lexicon: https://www.openslr.org/11, public domain.
• EmbeddingGemma 2 (google/embeddinggemma-2, Google DeepMind): Apache 2.0. It produces the soft tokens the reader reads and the word table. Deployments must adhere to the Gemma Prohibited Use Policy.