gte_L6_uniform_distilled (Distilled)

Lightweight sentence encoder created from alibaba-NLP/gte-multilingual-base via layer pruning + vocabulary pruning + knowledge distillation.

Model Details

Property	Value
Teacher	alibaba-NLP/gte-multilingual-base
Architecture	GTE-multilingual (pruned)
Hidden dim	768
Layers	6 / 12
Layer indices	[0, 2, 4, 7, 9, 11]
Strategy	6 layers, evenly spaced from GTE-multilingual (12L)
Parameters	234,919,680
Model size (FP32)	349.7MB
Distilled	Yes

Architecture

==============================================================
  TEACHER: GTE-multilingual  →  STUDENT: 6L / 63,531 vocab
==============================================================

            TEACHER                        STUDENT          
  ───────────────────────────    ───────────────────────────

  ┌─────────────────────────┐    ┌─────────────────────────┐
  │   Input Tokens          │    │   Input Tokens          │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Embeddings             │    │  Embeddings (pruned)    │
  │  vocab: 250,048         │    │  vocab:  63,531         │
  │  dim:  768              │    │  dim:  768              │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌─────────────────────────┐    ┌─────────────────────────┐
  │  Layer  0               │ ──►  │  Layer  0 ← L0         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  1               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  2               │ ──►  │  Layer  1 ← L2         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  3               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  4               │ ──►  │  Layer  2 ← L4         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  5               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  6               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  7               │ ──►  │  Layer  3 ← L7         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  8               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  9               │ ──►  │  Layer  4 ← L9         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 10               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 11               │ ──►  │  Layer  5 ← L11        │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Mean Pooling           │    │  Mean Pooling           │
  │  → 768d embedding       │    │  → 768d embedding       │
  └─────────────────────────┘    └─────────────────────────┘

  Size: 1058.2MB (FP32)           →  349.7MB (FP32)
  Params: 277,405,440        →  91,674,624
  Reduction: 67.0%
==============================================================

Quick Start

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("gte_L6_uniform_distilled", trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "안녕하세요",
    "Bonjour, comment allez-vous?",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 768)

MTEB Evaluation Results

Overall Average: 60.55%

Task Group	Average
Classification	65.06%
Clustering	39.6%
STS	75.15%

Classification

Task	Average	Details
AmazonCounterfactualClassification	72.89%	en-ext: 77.6%, en: 76.42%, de: 71.09%
Banking77Classification	84.6%	default: 84.6%
ImdbClassification	61.96%	default: 61.96%
MTOPDomainClassification	85.06%	en: 91.5%, es: 87.99%, fr: 84.93%
MassiveIntentClassification	42.29%	en: 72.77%, zh-CN: 70.33%, ja: 68.31%
MassiveScenarioClassification	47.31%	en: 77.55%, zh-CN: 76.14%, ja: 73.62%
ToxicConversationsClassification	65.05%	default: 65.05%
TweetSentimentExtractionClassification	61.34%	default: 61.34%

Clustering

Task	Average	Details
ArXivHierarchicalClusteringP2P	55.28%	default: 55.28%
ArXivHierarchicalClusteringS2S	50.15%	default: 50.15%
BiorxivClusteringP2P.v2	31.01%	default: 31.01%
MedrxivClusteringP2P.v2	32.96%	default: 32.96%
MedrxivClusteringS2S.v2	30.57%	default: 30.57%
StackExchangeClustering.v2	47.42%	default: 47.42%
StackExchangeClusteringP2P.v2	35.8%	default: 35.8%
TwentyNewsgroupsClustering.v2	33.61%	default: 33.61%

STS

Task	Average	Details
BIOSSES	73.46%	default: 73.46%
SICK-R	78.01%	default: 78.01%
STS12	77.31%	default: 77.31%
STS13	82.59%	default: 82.59%
STS14	80.24%	default: 80.24%
STS15	87.62%	default: 87.62%
STS17	67.67%	en-en: 86.24%, es-es: 82.71%, ko-ko: 74.85%
STS22.v2	45.07%	fr-pl: 73.25%, zh: 64.44%, es: 63.42%
STSBenchmark	84.39%	default: 84.39%

Distillation Impact

Task	Before	After	Delta
AmazonCounterfactualClassification	62.03%	72.89%	+10.86%p
ArXivHierarchicalClusteringP2P	53.65%	55.28%	+1.63%p
ArXivHierarchicalClusteringS2S	45.3%	50.15%	+4.85%p
Banking77Classification	58.65%	84.6%	+25.95%p
BiorxivClusteringP2P.v2	21.28%	31.01%	+9.73%p
BIOSSES	49.91%	73.46%	+23.55%p
ImdbClassification	63.58%	61.96%	-1.62%p
MassiveIntentClassification	30.15%	42.29%	+12.14%p
MassiveScenarioClassification	31.92%	47.31%	+15.39%p
MedrxivClusteringP2P.v2	26.07%	32.96%	+6.89%p
MedrxivClusteringS2S.v2	21.24%	30.57%	+9.33%p
MTOPDomainClassification	61.22%	85.06%	+23.84%p
SICK-R	51.42%	78.01%	+26.59%p
StackExchangeClustering.v2	39.07%	47.42%	+8.35%p
StackExchangeClusteringP2P.v2	32.7%	35.8%	+3.1%p
STS12	39.09%	77.31%	+38.22%p
STS13	51.12%	82.59%	+31.47%p
STS14	45.69%	80.24%	+34.55%p
STS15	60.2%	87.62%	+27.42%p
STS17	18.02%	67.67%	+49.65%p
STS22.v2	38.98%	45.07%	+6.09%p
STSBenchmark	54.35%	84.39%	+30.04%p
ToxicConversationsClassification	57.02%	65.05%	+8.03%p
TweetSentimentExtractionClassification	45.87%	61.34%	+15.47%p
TwentyNewsgroupsClustering.v2	10.91%	33.61%	+22.7%p

Training

Stage 1: Layer Pruning

Teacher: alibaba-NLP/gte-multilingual-base (12 layers, 768d)
Selected layers: [0, 2, 4, 7, 9, 11] (6 layers, evenly spaced from GTE-multilingual (12L))
Vocabulary pruning applied

Stage 2: Knowledge Distillation

Method: MSE + Cosine Similarity loss
Data: MTEB Classification/Clustering/STS task datasets
Optimizer: AdamW (lr=2e-5, weight_decay=0.01)
Schedule: Cosine annealing over 3 epochs

Supported Languages (18)

ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl

gomyk/gte-student-gte_L6_uniform_distilled