gemma_emb_compressed_distilled (Distilled)

Compact multilingual sentence encoder compressed from google/embeddinggemma-300m (24x compression).

Model Details

Quick Start

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("gemma_emb_compressed_distilled", trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "안녕하세요, 잘 지내세요?",
    "こんにちは、元気ですか？",
    "你好，你好吗？",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (4, 384)

MTEB Evaluation Results

Overall Average: 25.82%

Classification

Clustering

STS

Distillation Impact

Training

Stage 1: Model Compression

• Teacher: google/embeddinggemma-300m (24L, 768d, 303M params)
• Compression: Layer pruning → Hidden dim reduction → Vocab pruning
• Result: 4L / 384d / 19,485 vocab

Stage 2: Two-Stage Knowledge Distillation

Compression ratio 24x requires progressive distillation:

1. Stage 1: Teacher (303M) → Intermediate (~61M)
   • MSE + Cosine Similarity loss
   • MTEB task datasets (Classification/Clustering/STS)
2. Stage 2: Intermediate → Final Student (12.5M)
   • Same training objective
   • AdamW (lr=2e-5, weight_decay=0.01), Cosine annealing

License

This model is a derivative of Google's Gemma. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. Use of this model must comply with the Gemma Prohibited Use Policy.

Supported Languages (18)

ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl