ModernBERT-TR
Release blog · Model collection
A 150M-parameter Turkish encoder with an 8,192-token context window.
Model
ModernBERT-TR is a masked-language encoder based on the ModernBERT architecture with alternating global and local attention.
| Parameters | 149.4M |
| Context length | 8,192 tokens |
| Layers | 22 |
| Hidden width | 768 |
| Attention heads | 12 |
| GLU feed-forward width | 1,152 |
| Tokenizer | Cased WordPiece, 50,008 tokens |
Results
We evaluate ModernBERT-TR with task-specific fine-tuning on 28 TabiBench tasks, the fixed five-seed TrGLUE protocol, and frozen-encoder linear probing with encoder-fast-eval.
ModernBERT-TR leads the frozen-encoder evaluation and ranks second on TabiBench and TrGLUE.
Mean rank across 47 tasks grouped by capability; lower is better. ModernBERT-TR leads classification, pairwise inference, and retrieval, and ties TabiBERT on acceptability.
Usage
from transformers import pipeline
unmask = pipeline(
"fill-mask",
model="ytu-ce-cosmos/modernbert-tr-base",
)
unmask("Türkiye'nin başkenti [MASK]'dır.")
Training
We pretrain from random initialization with packed sequences, dynamic masking, BF16, and StableAdamW. Our effective batch size is 1.05M tokens.
| Phase | Training |
|---|---|
| 1k pretraining | 200B tokens at length 1,024; 6B-token warmup to 5e-4, then constant learning rate; masking 0.30 |
| 1k short decay | 20B-token continuation from the 1k endpoint; decay from 5e-4 to 0; masking 0.15 |
| 1k low-LR continuation | Rollback to about 40B tokens; 100B tokens at 1e-4, masking 0.10 |
| 1k decay | 1k decay soup from 1k low-LR continuation and 1k short decay |
| 8k context extension | 50B-token continuation from the 1k endpoint at length 8,192; constant 2e-4; masking 0.30 |
| 8k terminal decay | 20B-token continuation on the 8k mix; decay from 2e-4 to 0; masking 0.10 |
The 1k pretraining mix is about 81.4% Turkish, 8.5% English, 10.0% code, and 0.1% parallel text. The 8k mix adds long-form Turkish PDF, legal, and long-document sources.
Revisions
| Revision | Contents |
|---|---|
main |
ModernBERT-TR; 50% 1k decayed soup and 50% 8k terminal decay |
1k-no-decay |
Direct 200B-token 1k pretraining endpoint |
1k-decay |
50% 1k low-LR continuation and 50% 1k short decay |
8k-no-decay |
Direct 50B-token 8k context-extension endpoint |
License
Apache-2.0.