BrunoHays/embeddinggemma-2-nullspace-audio

BrunoHays/embeddinggemma-2-nullspace-audio is google/embeddinggemma-2 with no change to the frozen weights, plus a frozen 256-dimensional null-space head that only acts on audio inputs.

• Text, image and video inputs: identical to google/embeddinggemma-2 (same code path, same outputs).
• Audio inputs: the base 768d embedding is shifted along 256 directions that the text embedding space never uses (the null space of the terminal projection). The shift is computed from two frozen Conformer intermediate layers through a PCA projection fitted only on external, non-benchmark audio.
• No training. The head has no trainable parameters at inference time. Its frozen tensors are in nullspace_audio_head.safetensors (2.37 MB).

Architecture

For an audio clip, with the frozen audio tower of google/embeddinggemma-2:

1. Pool two Conformer layers over time (masked):
   • X₁ = [μ₂ ; σ₂] (2048d) from layer 2 (mean and std, then L2-normalised)
   • X₂ = μ₆ (1024d) from layer 6 (mean, then L2-normalised)
2. Project each block with its frozen PCA state (soft-whitened, fitted on external corpora):

   z_k = normalize((X_k - m_k) @ P_k) / (s_k + ε),   for k in {1, 2},  P_k shape: [d_k, 128]
   

   where s_k = σ_k^(1/2) is the stored (already square-rooted) scale and ε = 10⁻⁵.
3. Combine the two 128d blocks into a unit 256d vector:

   z   = normalize([√0.5 · z_1  ;  √0.5 · z_2])      shape: [256]
   

4. Inject into the base embedding v_base (768d, L2-normalised by the model) with γ = 1.0:

   v_final = normalize(v_base + γ · (z @ U_⊥^T))     shape: [768]
   

   where U_⊥ has shape [768, 256] with U_⊥^T @ U_⊥ = I.

Why Layers 2 (mean+std) and 6 (mean)

Consistent with layer-wise analyses of speech encoders (Pasad et al., 2021; Bouziane, Mdhaffar & Estève, 2026), different acoustic and paralinguistic attributes peak at different depths inside the 12-layer Conformer audio tower before the final text-aligned layers filter them out:

• Layer 2 (mean + std, 2048d → 128d): Early layers peak on speaker gender, timbre, pitch, and acoustic texture (VoxPopuliGenderClustering, IEMOCAPGender, MridinghamTonic). Including frame-level standard deviation (std) at Layer 2 captures temporal pitch and formant modulation.
• Layer 6 (mean, 1024d → 128d): Middle layers peak on vocal emotion (CREMA_D doubles from 19.7% at the final output to 39.5% at Layer 6). At middle and deeper layers, mean pooling alone outperforms mean + std because frame variance becomes noisy after L2 normalisation.

Why the null space

U_⊥ spans the orthogonal complement of the column space of the model's terminal projection W (768 × 512). Text embeddings lie in col(W), so they are orthogonal to U_⊥. The injection therefore does not move text embeddings, and it cannot change text-to-text similarities. Its effect is limited to audio embeddings, and cross-modal rankings move only slightly (see the MSEB section).

Because the 256 null-space directions are spread across all 768 coordinates and no retraining is done, audio embeddings must be used at the full 768d (Matryoshka truncation to 512/256/128 is not supported).

Zero-leakage PCA construction (Protocol A)

The PCA states are fitted only on 350 clips from each of four external corpora, with no MAEB or MSEB audio:

• ylacombe/expresso (speech)
• agkphysics/AudioSet (audio events)
• lewtun/music_genres (music)
• ashraq/esc50 (environmental sounds)

Clips are at most 10 s. The per-domain statistics are combined with balanced weights, then soft whitening (σ⁻¹/²) is applied on the same eigenbasis.

Usage

Run the standalone script with uv (installs dependencies automatically):

uv run https://huggingface.co/BrunoHays/embeddinggemma-2-nullspace-audio/resolve/main/embeddinggemma2_nullspace.py

Or from a local checkout:

uv run embeddinggemma2_nullspace.py

import numpy as np
from embeddinggemma2_nullspace import EmbeddingGemma2NullSpaceWrapper

model = EmbeddingGemma2NullSpaceWrapper()
audio_emb = model.encode_audio([np.zeros(16000, dtype=np.float32)])  # or ["speech.wav"]
text_emb = model.encode_text(["hello world"])
print(audio_emb.shape, text_emb.shape)  # (1, 768) (1, 768)

encode_audio accepts .wav file paths, 1D numpy/torch waveforms (16 kHz), (array, sr) tuples, or {"array": ..., "sampling_rate": ...} dicts.

With mteb (see MTEB PR #5775):

import mteb

model = mteb.get_model("BrunoHays/embeddinggemma-2-nullspace-audio")

The base checkpoint is loaded from google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b. Accepting the Gemma terms on the base model is required.

MAEB results (30 tasks)

On the MTEB Leaderboard (General Purpose -> Audio), the model reaches Rank #1 (Borda and Mean Task) on both MAEB(v1, audio-only) (jumping from #26 to #1 across 70 audio models) and full 30-task MAEB(v1) (jumping from #16 to #1):

"Mean (Task)" on MAEB(v1) evaluates FSD2019Kaggle on the curated subset per the MTEB benchmark definition (54.40% vs 49.31% base). "Raw" averages all subsets (55.09% vs 49.95% measured base).

Coverage: all 30 MAEB tasks were measured for this configuration. FleursT2ARetrieval (102 languages) and CommonVoiceMini21T2ARetrieval (117 languages) were run per language and macro-averaged over languages.

Per-task scores

Where it helps and where it hurts

The macro gain comes mainly from audio-to-audio and gender/emotion/genre tasks. Several tasks lose accuracy. The largest losses at γ = 1.0 are NMSQAPairClassification (−19.5 pp), SIBFLEURS (−35.9 pp), MInDS14 (−9.3 pp) and VoxCelebSA (−9.8 pp). Check these tasks before using the model where they matter.

MSEB results (Google SVQ, 26 locales)

Macro mean over 26 locales of google/svq (test split). Evaluated with the same head config, using mteb/svq. Measured in this pipeline; no other MSEB tasks were run.

The head moves retrieval scores by at most about 0.3 points on audio-to-text and about 0.05 points on text-to-audio. The effect is far smaller than the MAEB changes above.

Limitations

• Selection bias. The PCA construction is leak-free, but the choice of mode (ext_soft_whiten) and γ was made by looking at test scores across a small grid. Treat the MAEB numbers as optimistic. γ = 1.0 is the largest value tested for this mode, so the optimum may lie beyond it.
• Full 768d required. Because there is no retraining, audio embeddings require all 768 dimensions and do not support Matryoshka truncation.
• Single-run numbers. All 30 MAEB and 2 MSEB tasks were measured in this pipeline. The two multilingual T2A tasks were run per language; their scores are macro means over 102 and 117 languages.
• Mixed effects. Some audio tasks regress (see above). Do not assume the gain transfers to every audio task.
• Audio batching. In the audio path, clips are encoded in dataset order without SentenceTransformer's length sorting, so results can differ from the base model by about 10⁻³ per coordinate (bf16 padding effects).

License

Gemma license, inherited from google/embeddinggemma-2. Check the Gemma terms before redistributing.

Citation

If you use this model, please cite the base model:

@misc{vera2025embeddinggemmapowerfullightweighttext,
      title={EmbeddingGemma: Powerful and Lightweight Text Representations},
      author={Vera, Henrique Schechter and others},
      year={2025},
      eprint={2509.20354},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.20354},
}