modernbert_L4_uniform
Lightweight sentence encoder created from answerdotai/ModernBERT-base via layer pruning + vocabulary pruning.
Model Details
| Property | Value |
|---|---|
| Teacher | answerdotai/ModernBERT-base |
| Architecture | ModernBERT (pruned) |
| Hidden dim | 768 |
| Layers | 4 / 22 |
| Layer indices | [0, 7, 14, 21] |
| Strategy | 4 layers, evenly spaced from ModernBERT (22L) |
| Parameters | 55,607,040 |
| Model size (FP32) | 137.7MB |
| Distilled | No |
Architecture
==============================================================
TEACHER: ModernBERT → STUDENT: 4L / 24,978 vocab
==============================================================
TEACHER STUDENT
─────────────────────────── ───────────────────────────
┌─────────────────────────┐ ┌─────────────────────────┐
│ Input Tokens │ │ Input Tokens │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌────────────┴────────────┐ ┌────────────┴────────────┐
│ Embeddings │ │ Embeddings (pruned) │
│ vocab: 50,368 │ │ vocab: 24,978 │
│ dim: 768 │ │ dim: 768 │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌─────────────────────────┐ ┌─────────────────────────┐
│ Layer 0 │ ──► │ Layer 0 ← L0 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 1 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 2 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 3 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 4 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 5 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 6 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 7 │ ──► │ Layer 1 ← L7 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 8 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 9 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 10 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 11 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 12 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 13 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 14 │ ──► │ Layer 2 ← L14 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 15 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 16 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 17 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 18 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 19 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 20 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 21 │ ──► │ Layer 3 ← L21 │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌────────────┴────────────┐ ┌────────────┴────────────┐
│ Mean Pooling │ │ Mean Pooling │
│ → 768d embedding │ │ → 768d embedding │
└─────────────────────────┘ └─────────────────────────┘
Size: 495.8MB (FP32) → 137.7MB (FP32)
Params: 129,980,160 → 36,107,520
Reduction: 72.2%
==============================================================
Quick Start
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("modernbert_L4_uniform", trust_remote_code=True)
sentences = [
"Hello, how are you?",
"안녕하세요",
"Bonjour, comment allez-vous?",
]
embeddings = model.encode(sentences)
print(embeddings.shape) # (3, 768)
MTEB Evaluation Results
Overall Average: 39.64%
| Task Group | Average |
|---|---|
| Classification | 46.21% |
| Clustering | 27.69% |
| STS | 44.42% |
Classification
| Task | Average | Details |
|---|---|---|
| AmazonCounterfactualClassification | 62.15% | en: 65.7%, de: 62.22%, en-ext: 61.47% |
| Banking77Classification | 45.76% | default: 45.76% |
| ImdbClassification | 57.29% | default: 57.29% |
| MTOPDomainClassification | 49.25% | es: 53.78%, en: 53.67%, de: 49.66% |
| MassiveIntentClassification | 29.75% | zh-CN: 42.91%, ja: 38.38%, zh-TW: 37.92% |
| MassiveScenarioClassification | 30.03% | zh-CN: 45.57%, zh-TW: 38.84%, en: 37.16% |
| ToxicConversationsClassification | 55.82% | default: 55.82% |
| TweetSentimentExtractionClassification | 39.6% | default: 39.6% |
Clustering
| Task | Average | Details |
|---|---|---|
| ArXivHierarchicalClusteringP2P | 47.97% | default: 47.97% |
| ArXivHierarchicalClusteringS2S | 45.05% | default: 45.05% |
| BiorxivClusteringP2P.v2 | 15.49% | default: 15.49% |
| MedrxivClusteringP2P.v2 | 23.89% | default: 23.89% |
| MedrxivClusteringS2S.v2 | 18.83% | default: 18.83% |
| StackExchangeClustering.v2 | 32.31% | default: 32.31% |
| StackExchangeClusteringP2P.v2 | 29.14% | default: 29.14% |
| TwentyNewsgroupsClustering.v2 | 8.87% | default: 8.87% |
STS
| Task | Average | Details |
|---|---|---|
| BIOSSES | 42.22% | default: 42.22% |
| SICK-R | 54.53% | default: 54.53% |
| STS12 | 40.85% | default: 40.85% |
| STS13 | 46.53% | default: 46.53% |
| STS14 | 48.23% | default: 48.23% |
| STS15 | 61.72% | default: 61.72% |
| STS17 | 28.67% | es-es: 67.31%, en-en: 63.62%, ko-ko: 53.02% |
| STS22.v2 | 28.6% | zh: 60.37%, es: 54.2%, it: 51.29% |
| STSBenchmark | 48.43% | default: 48.43% |
Training
Created via layer pruning + vocabulary pruning (no additional training):
- Teacher:
answerdotai/ModernBERT-base(22 layers, 768d) - Layer selection:
[0, 7, 14, 21]- 4 layers, evenly spaced from ModernBERT (22L) - Vocab pruning: Corpus-based filtering for target languages
Supported Languages (18)
ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl