Mizan-Embeddings-0.1B

Mizan-Embeddings-0.1B is a compact Arabic text embedding model for semantic search and retrieval. It maps sentences, passages and long documents (up to 8,192 tokens) to a 768-dimensional vector space.

It is the smaller member of the Mizan-Embeddings family. For higher accuracy, see Mizan-Embeddings-0.4B.

Model Details

• Model Type: Sentence Transformer (bi-encoder)
• Architecture: ModernBERT-base (22 layers, hidden size 768, 12 heads, alternating local 128-token / global attention, RoPE)
• Parameters: 149M
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768
• Pooling: mean pooling
• Similarity Function: cosine similarity
• Prompts / instructions: none (queries and documents are encoded the same way)
• Language: Arabic
• License: apache-2.0

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ALJIACHI/Mizan-Embeddings-0.1B")

queries = ["ما هي فوائد شرب الماء للجسم؟"]
documents = [
    "يساعد شرب كميات كافية من الماء على تنظيم حرارة الجسم وتحسين الهضم والحفاظ على صحة الكلى.",
    "تحتوي الخضروات الورقية على نسبة عالية من الألياف والفيتامينات المفيدة للجسم.",
    "تعد القاهرة من أكبر المدن في القارة الأفريقية من حيث عدد السكان.",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (1, 768) (3, 768)

similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7327, 0.4321, 0.0326]])

For long documents keep the default max_seq_length of 8192; for short passages you can lower it (model.max_seq_length = 512) to speed up encoding. Flash Attention 2 is recommended on GPU:

import torch
model = SentenceTransformer(
    "ALJIACHI/Mizan-Embeddings-0.1B",
    model_kwargs={"torch_dtype": torch.float16, "attn_implementation": "flash_attention_2"},
)

Evaluation

Mizan-Embeddings-0.1B compared with embedding models of similar size (107M-149M parameters) that support Arabic. The larger Mizan-Embeddings-0.4B is included as a reference; bold marks the best score among the 0.1B-class models.

All scores are on a 0-1 scale; Average is the unweighted mean of the three columns.

• Islamic & Arabic Retrieval: mean nDCG@10 over three in-house Arabic retrieval sets: ranking candidate rulings for Islamic jurisprudence (fiqh) questions, retrieving them from the full pool of ~5.2k rulings, and Arabic open-domain question answering over ~4.6k passages. These sets were used for model selection during fine-tuning, so this column is slightly optimistic for the Mizan models.
• Open-Domain Retrieval: mean nDCG@10 over three public Arabic test sets whose queries were never used for training or model selection: Qur'an question answering (IslamicEval 2025, 1,266 passages), native Arabic questions over Wikipedia (SadeemQuestionRetrieval, 23k passages), and long-document retrieval (MLDR Arabic test, 7.6k articles encoded at up to 8,192 tokens).
• Arabic MTEB (3 tasks): mean of the official MTEB main scores on AJGT.v2 (classification, accuracy), STS17 ar-ar (semantic similarity, Spearman) and PublicHealthQA Arabic (retrieval, nDCG@10).

Framework Versions

• Sentence Transformers 5.4.1, Transformers 4.55.4, PyTorch 2.8.0, Tokenizers 0.21.0, Flash Attention 2.8.3

Limitations

• Trained mostly on Modern Standard Arabic: dialectal and social-media Arabic (e.g. tweets) is weaker than formal text.
• Arabic only; it was not trained for other languages or cross-lingual retrieval.
• On short-sentence similarity (STS) and sentiment classification, Arabic sentence-similarity models of the same size can score higher; Mizan-Embeddings-0.1B is optimized for retrieval.

Citation

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

@misc{warner2024modernbert,
    title = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
    author = {Benjamin Warner and Antoine Chaffin and Benjamin Clavi{\'e} and Orion Weller and Oskar Hallstr{\"o}m and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
    year = {2024},
    eprint = {2412.13663},
    archivePrefix = {arXiv},
}

@misc{gao2021scaling,
    title = {Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author = {Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year = {2021},
    eprint = {2101.06983},
    archivePrefix = {arXiv},
}