Mizan-Embeddings-0.4B

Mizan-Embeddings-0.4B is an Arabic text embedding model for semantic search and retrieval. It maps sentences, passages and long documents (up to 8,192 tokens) to a 1024-dimensional vector space.

It is the larger member of the Mizan-Embeddings family (after Mizan-Embeddings-0.1B): a 28-layer ModernBERT encoder that was first pre-trained from scratch on Arabic text with an Arabic tokenizer, then fine-tuned for retrieval in two contrastive stages.

Model Details

• Model Type: Sentence Transformer (bi-encoder)
• Architecture: ModernBERT-large (28 layers, hidden size 1024, 16 heads, alternating local 128-token / global attention, RoPE)
• Parameters: 395M
• Tokenizer: Arabic byte-level BPE, 50,280 tokens
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024
• Pooling: mean pooling
• Similarity Function: cosine similarity
• Prompts / instructions: none (queries and documents are encoded the same way)
• Language: Arabic
• License: apache-2.0

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ALJIACHI/Mizan-Embeddings-0.4B")

queries = ["ما هي فوائد شرب الماء للجسم؟"]
documents = [
    "يساعد شرب كميات كافية من الماء على تنظيم حرارة الجسم وتحسين الهضم والحفاظ على صحة الكلى.",
    "تحتوي الخضروات الورقية على نسبة عالية من الألياف والفيتامينات المفيدة للجسم.",
    "تعد القاهرة من أكبر المدن في القارة الأفريقية من حيث عدد السكان.",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (1, 1024) (3, 1024)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7207, 0.2488, 0.0833]])

For long documents keep the default max_seq_length of 8192; for short passages you can lower it (model.max_seq_length = 512) to speed up encoding. Flash Attention 2 is recommended on GPU:

import torch
model = SentenceTransformer(
    "ALJIACHI/Mizan-Embeddings-0.4B",
    model_kwargs={"torch_dtype": torch.float16, "attn_implementation": "flash_attention_2"},
)

Evaluation

All scores are on a 0-1 scale; Average is the unweighted mean of the three columns.

• Islamic & Arabic Retrieval: mean nDCG@10 over three in-house Arabic retrieval sets: ranking candidate rulings for Islamic jurisprudence (fiqh) questions, retrieving them from the full pool of ~5.2k rulings, and Arabic open-domain question answering over ~4.6k passages. These sets were used for model selection during fine-tuning, so this column is slightly optimistic for the Mizan models.
• Open-Domain Retrieval: mean nDCG@10 over three public Arabic test sets whose queries were never used for training or model selection: Qur'an question answering (IslamicEval 2025, 1,266 passages), native Arabic questions over Wikipedia (SadeemQuestionRetrieval, 23k passages), and long-document retrieval (MLDR Arabic test, 7.6k articles encoded at up to 8,192 tokens).
• Arabic MTEB (3 tasks): mean of the official MTEB main scores on AJGT.v2 (classification, accuracy), STS17 ar-ar (semantic similarity, Spearman) and PublicHealthQA Arabic (retrieval, nDCG@10).

Arabic MTEB (×100)

All models were run by us with the same code, at max_seq_length min(model limit, 8192) (embeddinggemma-300m: 2048), using each model's recommended query / document prompts.

Training

Training ran in three phases.

1. Arabic pre-training (masked language modelling)

Starting from ModernBERT-large with an Arabic tokenizer, the model was pre-trained on 20B tokens of Arabic web text: 18B tokens at 1,024-token sequences, then 2B tokens of long documents at 8,192 tokens to extend the context window.

2. Contrastive stage 1: general Arabic semantics

• Data: ~916k Arabic (anchor, positive, negative) triplets (~97M tokens): natural language inference, paraphrase and question-passage pairs; pairs overlapping our in-house evaluation sets or the Arabic MTEB STS / NLI test sets were removed.
• Objective: multiple-negatives ranking loss (InfoNCE) with in-batch negatives plus one hard negative per anchor, batch size 256 (GradCache), sequences up to 256 tokens.
• Schedule: learning rate 2e-5, 5% linear warmup.

3. Contrastive stage 2: long-context retrieval

• Data: ~524k Arabic query-document pairs (~0.67B tokens), each with 3 hard negatives mined with a dense retriever and filtered with a cross-encoder reranker to remove false negatives. Domains: Islamic jurisprudence questions and rulings, hadith, religious Q&A, encyclopedic question answering, general-domain QA and long-document retrieval (documents up to 8,192 tokens), plus a small share of NLI / paraphrase pairs.
• Objective: multiple-negatives ranking loss with GradCache, batch size 256; each batch comes from a single source so in-batch negatives are same-domain, with no duplicate texts in a batch.
• Schedule: learning rate 2e-5, 5% warmup, linear decay.

Framework Versions

• Sentence Transformers 5.4.1, Transformers 4.55.4, PyTorch 2.8.0, Tokenizers 0.21.0, Flash Attention 2.8.3

Limitations

• Trained mostly on Modern Standard Arabic: dialectal and social-media Arabic (e.g. tweets) is weaker than formal text.
• Arabic only; it was not trained for other languages or cross-lingual retrieval.
• 1024-dimensional vectors and ~2.5x the compute of Mizan-Embeddings-0.1B per text: re-index existing vector stores when switching models.

Citation

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

@misc{warner2024modernbert,
    title = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
    author = {Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
    year = {2024},
    eprint = {2412.13663},
    archivePrefix = {arXiv},
}

@misc{gao2021scaling,
    title = {Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author = {Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year = {2021},
    eprint = {2101.06983},
    archivePrefix = {arXiv},
}