FADA: Arabic text embeddings

_{Full-resolution poster (4K PNG)}

FADA maps Arabic text to a 768-dimensional vector. It is a Matryoshka model: the first 512, 256, 128, 64 dimensions can be used on their own with little loss. It was built in two stages:

1. Contrastive fine-tuning. UBC-NLP/ARBERTv2 fine-tuned on Arab3M-Triplets with an in-batch contrastive (InfoNCE) loss and Matryoshka representation learning.
2. Distillation. That model was then trained to reproduce the embeddings of Qwen/Qwen3-Embedding-8B on the texts of the same dataset.

The stage-1 recipe is inspired by Swan-Small (Bhatia et al., 2024), which also fine-tunes ARBERTv2 with InfoNCE. This is an independent implementation on different data.

Results

Semantic textual similarity on Arabic, from MTEB (mteb==2.22.0): STS17 (ar-ar) and STS22.v2 (ar). Score = Spearman correlation x 100 between the model's cosine similarity and human ratings. Higher is better. Evaluated with max_seq_length=512.

Training

Stage 1: contrastive fine-tuning

Stage 2: distillation from Qwen/Qwen3-Embedding-8B

Distillation loss

kd_cos and kd_sim are the two loss terms after weighting (x10 and x200), each averaged over the 5 Matryoshka sizes; the values are averages over each 25-step logging window.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("masterofaudio2077/Fada_ar_embedding")                      # 768 dims
# model = SentenceTransformer("masterofaudio2077/Fada_ar_embedding", truncate_dim=256)  # Matryoshka: 512, 256, 128 or 64 also work

sentences = ["ما أجمل الليل الهادئ", "سماء الليل مليئة بالنجوم", "أحب القهوة في الصباح"]
embeddings = model.encode(sentences)
print(model.similarity(embeddings, embeddings))

Without Sentence-Transformers (plain transformers, mean pooling over the tokens; slice the first d numbers for a smaller Matryoshka size):

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("masterofaudio2077/Fada_ar_embedding")
model = AutoModel.from_pretrained("masterofaudio2077/Fada_ar_embedding").eval()

def embed(texts, dim=768):
    batch = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        hidden = model(**batch).last_hidden_state
    mask = batch["attention_mask"].unsqueeze(-1).float()
    pooled = (hidden * mask).sum(1) / mask.sum(1)
    return pooled[:, :dim]

vecs = embed(["ما أجمل الليل الهادئ", "سماء الليل مليئة بالنجوم"])
print(torch.nn.functional.cosine_similarity(vecs[0], vecs[1], dim=0))

Notes:

• No query or passage prefix is needed (none was used in training).
• The output vectors are not L2-normalised. Cosine similarity (the default of model.similarity) is unaffected; use normalize_embeddings=True if you need unit vectors.
• The saved default max_seq_length is 200; set model.max_seq_length = 512 for longer texts, as in the benchmark above.
• The weights are stored as model.safetensors. The repository was saved with sentence-transformers 6.1.0; if loading fails on an older version, run pip install -U sentence-transformers.

Architecture:

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

License and terms

Released under apache-2.0. Please note: the base model ARBERTv2 states no license on its model page, and the training dataset Arab3M-Triplets is labelled Apache-2.0 on the Hub but its access form asks users to agree to non-commercial use only. The teacher Qwen3-Embedding-8B is Apache-2.0. Check these terms before any commercial use.

Citations

If you use this model, please also cite the work it builds on. The training data (Arab3M-Triplets) provides no paper or BibTeX entry; cite it by name and link.

@inproceedings{abdul-mageed-etal-2021-arbert,
    title = "{ARBERT} {\&} {MARBERT}: Deep Bidirectional Transformers for {A}rabic",
    author = "Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and Nagoudi, El Moatez Billah",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = aug, year = "2021", address = "Online", publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-long.551", doi = "10.18653/v1/2021.acl-long.551", pages = "7088--7105"
}

@article{elmadany2022orca,
  title={ORCA: A Challenging Benchmark for Arabic Language Understanding},
  author={Elmadany, AbdelRahim and Nagoudi, El Moatez Billah and Abdul-Mageed, Muhammad},
  journal={arXiv preprint arXiv:2212.10758}, year={2022}
}

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}

@misc{zhang2025jasperstelladistillationsota,
  title={Jasper and Stella: distillation of SOTA embedding models},
  author={Dun Zhang and Jiacheng Li and Ziyang Zeng and Fulong Wang},
  year={2025}, eprint={2412.19048}, archivePrefix={arXiv}, primaryClass={cs.IR},
  url={https://arxiv.org/abs/2412.19048}
}

@article{bhatia2024swan,
  title={Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarks},
  author={Bhatia, Gagan and Nagoudi, El Moatez Billah and El Mekki, Abdellah and Alwajih, Fakhraddin and Abdul-Mageed, Muhammad},
  journal={arXiv preprint arXiv:2411.01192}, year={2024}
}

@article{kusupati2022matryoshka,
  title={Matryoshka Representation Learning},
  author={Kusupati, Aditya and Bhatt, Gantavya and Rege, Aniket and Wallingford, Matthew and Sinha, Aditya and Ramanujan, Vivek and Howard-Snyder, William and Chen, Kaifeng and Kakade, Sham and Jain, Prateek and Farhadi, Ali},
  journal={arXiv preprint arXiv:2205.13147}, year={2022}
}

@inproceedings{gao2021scaling,
  title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
  author={Gao, Luyu and Zhang, Yunyi and Han, Jiawei and Callan, Jamie},
  booktitle={RepL4NLP 2021}, year={2021}, note={arXiv:2101.06983}
}

@article{oord2018representation,
  title={Representation Learning with Contrastive Predictive Coding},
  author={van den Oord, Aaron and Li, Yazhe and Vinyals, Oriol},
  journal={arXiv preprint arXiv:1807.03748}, year={2018}
}

@article{chen2023lion,
  title={Symbolic Discovery of Optimization Algorithms},
  author={Chen, Xiangning and Liang, Chen and Huang, Da and Real, Esteban and Wang, Kaiyuan and Liu, Yao and Pham, Hieu and Dong, Xuanyi and Luong, Thang and Hsieh, Cho-Jui and Lu, Yifeng and Le, Quoc V.},
  journal={arXiv preprint arXiv:2302.06675}, year={2023}
}

@inproceedings{loshchilov2019decoupled,
  title={Decoupled Weight Decay Regularization},
  author={Loshchilov, Ilya and Hutter, Frank},
  booktitle={International Conference on Learning Representations}, year={2019}, note={arXiv:1711.05101}
}

@inproceedings{reimers2019sentence,
  title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
  author={Reimers, Nils and Gurevych, Iryna},
  booktitle={EMNLP 2019}, year={2019}, note={arXiv:1908.10084}
}

@article{muennighoff2022mteb,
  title={MTEB: Massive Text Embedding Benchmark},
  author={Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"i}c and Reimers, Nils},
  journal={arXiv preprint arXiv:2210.07316}, year={2022}
}

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672}, year={2024}
}

Dataset: Omartificial-Intelligence-Space, Arab3M-Triplets, https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arab3M-Triplets

Card generated on 2026-10-02.