Arabic ModernBERT Unified

A compact Arabic text encoder with 32.79M parameters for masked-language modeling and 32.94M with all task heads. Its eight-layer ModernBERT backbone supports masked-word prediction, three-way natural language inference, extractive QA, and 384-dimensional sentence embeddings.

Quick start

Use the standard Transformers fill-mask pipeline. Tested with Python 3.13, PyTorch 2.13.0 and Transformers 5.14.1.

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="Abdelkareem/arabic-modernbert-unified")
for candidate in fill_mask("تقع القاهرة على ضفاف نهر [MASK].", top_k=5):
    print(candidate["token_str"], candidate["score"])

Sentence embeddings and NLI

Download the release and run the included example.

hf download Abdelkareem/arabic-modernbert-unified --revision v1.0.0 --local-dir arabic-modernbert-unified
cd arabic-modernbert-unified
pip install -r requirements.txt
python example.py

from unified_encoder import UnifiedArabicEncoder

model = UnifiedArabicEncoder(".")
embeddings = model.encode(["يقرأ الطفل كتاباً.", "الطفل يقرأ كتاباً."])
print(float(embeddings[0] @ embeddings[1]))

scores = model.classify(
    ["يقرأ الطفل كتاباً في المكتبة."],
    ["الطفل يقرأ كتاباً."],
)[0]
print(dict(zip(("entailment", "neutral", "contradiction"), scores.tolist())))

The retrieval path mean-pools non-special tokens, applies the learned projection and L2-normalizes each vector. NLI uses the first-token hidden state and the released three-way classifier. The QA head is accessible as model.qa and maps each hidden state to start and end logits. For extractive QA, encode the question as the first sequence and the context as the second, and use only context offsets for answer spans.

Training

1. Arabic pretraining. A single pass over the retained Arabic arb_Arab subset of FineWeb2-HQ, pinned at c0c06e94fd3a44ae9e802b2b0fc533817601eb5e. After cleanup, overlap exclusions and exact deduplication, the run retained 3,098,999 documents and 18.30 GB of cleaned UTF‑8 text. The model saw 2.508B content tokens and 2.517B attended positions in 142,210 updates using 15% token masking. FineWeb2-HQ dataset card
2. Language and task continuation. The selected lineage used merged Arabic Wikipedia, HQ and Arabic XL-Sum language replay, Arabic NLI, TyDi QA and filtered domain examples. The reconstructed continuation budget is ~778M attended positions. Replay contributes to this count.
3. Unified adaptation. The final 1,000 updates combined MLM, NLI with teacher supervision, contrastive retrieval with teacher-vector alignment, and extractive QA replay. Domain MLM included Nawah and filtered Waqfeya text. Training manifest

Evaluation

Reviewed scores show Luna → Terra percentages on 100 shared Arabic cases per task. MLM credits exact answers and plausible completions accepted by the judge. NLI measures judge agreement. Retrieval measures whether the top results include a judged relevant sentence. All checkpoints used the same frozen probe protocol for NLI and retrieval.

Unified uses 78% fewer MLM parameters than NAMAA. On these cases it comes within two points of NAMAA on sentence retrieval, while NAMAA leads by 13–14 points on MLM top‑1. Unified leads all three public baselines on NLI agreement after its own supervised task adaptation. The public checkpoints' encoder weights remained frozen.

Unified uses 34% fewer parameters for MLM than ArabicBertHash v2 and scores higher on all five reported metrics under both judges. ArabicBertHash v2 has a 16.25M-parameter encoder and a larger, untied MLM head. Its encoder is configured for 2,048 tokens, with a 1,024-token tokenizer limit in the release. Its NLI and retrieval scores come from the shared probes.

Exact-reference and native-head results

Exact MLM scores require the original masked word. A blinded spot review found that each automated judge missed valid alternatives and accepted some invalid answers. The MLM scorer rejects blank and control-token candidates. Machine-readable evaluation

Scope and limitations

These diagnostics use short examples from previously used evaluation sets. Retrieval uses NLI-derived sentence pairs, and the NLI comparison includes the benefit of Unified’s prior supervised training. Long-document and extractive QA performance remain unmeasured here.

Files

Architecture reference: ModernBERT. The Arabic checkpoint was initialized and trained from scratch.