Mursit-Large-TR-Retrieval

Model Description

Mursit-Large-TR-Retrieval is a large-scale Turkish embedding model pre-trained entirely from scratch on Turkish-dominant corpora and fine-tuned for retrieval tasks. The model is based on ModernBERT-large architecture (403M parameters) and optimized specifically for Turkish legal domain applications. This model achieves strong performance on Turkish retrieval benchmarks with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.

Key Features:

• Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
• Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
• Achieves strong performance on Turkish legal retrieval benchmarks
• Optimized for Turkish legal domain with custom tokenizer trained on legal documents

Model Type: Embedding
Parameters: 403M
Base Model: newmindai/Mursit-Large
Architecture: ModernBERT-large
Embedding Dimension: 1,024
Max Sequence Length: 2,048 tokens

Architecture Details

The model is based on ModernBERT-large architecture:

• Attention Mechanism: Alternating local and global attention
• Normalization: Pre-layer normalization with RMSNorm
• Activation: GeGLU (Gated Linear Units with GELU) in MLP layers
• Position Embeddings: Rotary positional embeddings (RoPE) with θ=20,000
• Context Length: 2,048 tokens
• Layers: 28 transformer layers
• Hidden Size: 1,024
• FFN Size: 2,624
• Attention Heads: 16 heads with 64 dimensions each
• Window Size: 128 (for sliding window attention in local layers)
• Vocabulary Size: 59,008 tokens

Training Details

Pre-training:

• Dataset: Turkish-dominant corpus totaling approximately 112.7 billion tokens
  • Legal Sources:
    • Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
    • Council of State (Danıştay): 151K sequences, ~0.11B tokens
    • Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
  • General Turkish Sources:
    • FineWeb2: General Turkish web data
    • CulturaX: Multilingual corpus (Turkish subset)
    • Total general Turkish: 212M sequences, ~96.17B tokens
  • Data Processing: SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
• Training Method: Masked Language Modeling (MLM) with 15% masking probability
• Masking Strategy: 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
• Framework: MosaicML Composer with Decoupled StableAdamW optimizer
• Learning Rate: 8×10⁻⁴ with warmup_stable_decay schedule
• Precision: BF16 mixed precision
• Hardware Infrastructure:
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Compute Nodes: 32 nodes
  • GPUs: 128× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
  • Node Configuration: Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
  • Interconnect: 800 Gb/s InfiniBand for distributed training
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
  • Distributed Training: Multi-node distributed training across 32 nodes with InfiniBand interconnect

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
• Training Framework: Sentence Transformers
• Optimization: Contrastive learning on Turkish passage ranking dataset
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
• Optimizer: AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Large-TR-Retrieval achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.

Column abbreviations: MTEB = mean performance across task types; Legal = weighted average of Contracts, Regulation, Caselaw; Classification = accuracy on Turkish classification tasks; Clustering = V-measure on clustering tasks; Pair Classification = average precision on pair classification tasks like NLI; Retrieval = nDCG@10 on information retrieval tasks; Semantic Textual Similarity = Spearman correlation; Contracts = nDCG@10 on legal contract retrieval; Regulation = nDCG@10 on regulatory text retrieval; Caselaw = nDCG@10 on case law retrieval; Number of Parameters = number of model parameters; Model Type = model type (Embedding, CLM-Embedding, Masked Language Model). Bold values indicate the highest score in each column.

Key Findings:

• The model achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models
• Strong performance on Contracts retrieval (81.78 nDCG@10) demonstrates effectiveness for legal document search
• Post-training on MS MARCO-TR significantly improves retrieval capabilities compared to base MLM models

Post-Training Performance Analysis

The following visualization shows the impact of post-training on retrieval performance:

Post-Training Retrieval Performance Comparison. Post-trained models (Mursit-Base-TR-Retrieval and Mursit-Large-TR-Retrieval) show significant improvements in legal domain retrieval tasks compared to base MLM models.

Reproducibility

To reproduce the benchmark results and training procedures for this model, please refer to:

• Post-Training: github.com/newmindai/mecellem-models/training/post-training-retrieval - Contains code and configurations for post-training retrieval models on MS MARCO-TR dataset.
• Embedding Benchmark Results: github.com/newmindai/mecellem-models/benchmark/embedding_model - Contains code and evaluation configurations for reproducing MTEB-Turkish benchmark results.

Usage

Installation

pip install sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("newmindai/Mursit-Large-TR-Retrieval")

# Encode sentences
sentences = [
    "Türk hukuk sistemi medeni hukuk geleneğine dayanır",
    "Anayasa Türkiye Cumhuriyeti'nin temel hukuk belgesidir"
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (2, 1024)

Information Retrieval

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("newmindai/Mursit-Large-TR-Retrieval")

query = "Sözleşme feshi nasıl yapılır?"
documents = [
    "Sözleşmeler yazılı olarak feshedilebilir.",
    "İş kanunu çalışma koşullarını düzenler."
]

query_embedding = model.encode(query, convert_to_tensor=True)
doc_embeddings = model.encode(documents, convert_to_tensor=True)
scores = util.cos_sim(query_embedding, doc_embeddings)[0]

results = [(doc, score.item()) for doc, score in zip(documents, scores)]
results.sort(key=lambda x: x[1], reverse=True)

for doc, score in results:
    print(f"Score: {score:.4f} - {doc}")

ONNX Model Inference

This script demonstrates how to use the ONNX model from Hugging Face for text embedding generation.

Exporting Model to ONNX

To export the model to ONNX format, use the optimum-cli command:

optimum-cli export onnx \
  -m newmindai/Mursit-Large-TR-Retrieval \
  --task feature-extraction \
  onnx/MursitLargeTRRetrieval

This will create the model.onnx file in the specified output directory.

Installation

pip install onnxruntime-gpu transformers huggingface_hub numpy

Usage

import onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
import numpy as np

model_id = "newmindai/Mursit-Large-TR-Retrieval"

# Load tokenizer and download ONNX model from Hugging Face
tokenizer = AutoTokenizer.from_pretrained(model_id)
onnx_path = hf_hub_download(repo_id=model_id, filename="model.onnx")

# Use GPU if available, otherwise fallback to CPU
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"] if "CUDAExecutionProvider" in ort.get_available_providers() else ["CPUExecutionProvider"]

sess = ort.InferenceSession(onnx_path, providers=providers)

texts = ["This is a test"]
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="np")

outputs = sess.run(None, {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64),
})

embeddings = outputs[-1]  # sentence_embedding is usually the last output
print(embeddings.shape)
print(embeddings[:1])

Features

• Automatic GPU/CPU selection: Uses CUDA if available, otherwise falls back to CPU
• Hugging Face integration: Downloads model files directly from Hugging Face Hub
• Simple API: Easy-to-use interface for text embedding generation

Use Cases

• Semantic search in Turkish legal documents
• Legal document retrieval and ranking
• Contract similarity and matching
• Regulation compliance checking
• Case law research and discovery
• Question answering systems for legal domain

Reproducibility

To reproduce the benchmark results and training procedures for this model, please refer to:

• Post-Training: github.com/newmindai/mecellem-models/training/post-training-retrieval - Contains code and configurations for post-training retrieval models on MS MARCO-TR dataset.
• Embedding Benchmark Results: github.com/newmindai/mecellem-models/benchmark/embedding_model - Contains code and evaluation configurations for reproducing MTEB-Turkish benchmark results.

Acknowledgments

This work was supported by the EuroHPC Joint Undertaking through project etur46 with access to the MareNostrum 5 supercomputer, hosted by Barcelona Supercomputing Center (BSC), Spain. MareNostrum 5 is owned by EuroHPC JU and operated by BSC. We are grateful to the BSC support team for their assistance with job scheduling, environment configuration, and technical guidance throughout the project.

The numerical calculations reported in this work were fully/partially performed at TÜBİTAK ULAKBİM, High Performance and Grid Computing Center (TRUBA resources). The authors gratefully acknowledge the know-how provided by the MINERVA Support for expert guidance and collaboration opportunities in HPC-AI integration.

References

If you use this model, please cite our paper:

@article{mecellem2026,
  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
  author={Uğur, Özgür and Göksu, Mahmut and Çimen, Mahmut and Yılmaz, Musa and Şavirdi, Esra and Demir, Alp Talha and Güllüce, Rumeysa and Çetin, İclal and Sağbaş, Ömer Can},
  journal={arXiv preprint arXiv:2601.16018},
  year={2026},
  month={January},
  url={https://arxiv.org/abs/2601.16018},
  doi={10.48550/arXiv.2601.16018},
  eprint={2601.16018},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Base Model References

@inproceedings{modernbert2025,
  title={ModernBERT: A Modern Bidirectional Encoder Transformer},
  author={Answer.AI and LightOn},
  booktitle={Proceedings of the 2025 Conference on Language Models},
  year={2025}
}

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation}, 
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}