Mursit-Base-TR-Retrieval

Model Description

Mursit-Base-TR-Retrieval is a Turkish embedding model pre-trained entirely from scratch on Turkish-dominant corpora and fine-tuned for retrieval tasks. The model is based on ModernBERT-base architecture (155M parameters) and optimized specifically for Turkish legal domain applications. This model demonstrates that trainable Masked Language Modeling (MLM) models can effectively serve as foundations for embedding tasks when training quality is assessed through downstream performance rather than MLM loss minimization alone.

Key Features:

• Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
• Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
• Achieves strong performance on Turkish legal retrieval benchmarks (55.86 MTEB Score, 47.52 Legal Score)
• Optimized for Turkish legal domain with custom tokenizer trained on legal documents

Model Type: Embedding
Parameters: 155M
Base Model: newmindai/Mursit-Base
Architecture: ModernBERT-base
Embedding Dimension: 768
Max Sequence Length: 1,024 tokens

Architecture Details

The model is based on ModernBERT architecture, which incorporates modern architectural advances for bidirectional encoders:

• Attention Mechanism: Alternating local and global attention to efficiently handle long contexts
• Normalization: Pre-layer normalization with RMSNorm
• Activation: GeGLU (Gated Linear Units with GELU) in MLP layers
• Position Embeddings: Rotary positional embeddings (RoPE) with θ=10,000
• Context Length: 1,024 tokens
• Layers: 22 transformer layers
• Hidden Size: 768
• FFN Size: 1,152
• Attention Heads: 12 heads with 64 dimensions each
• Window Size: 128 (for sliding window attention in local layers)
• Vocabulary Size: 59,008 tokens

The model uses a custom tokenizer trained on Turkish web data and legal documents, employing Byte Pair Encoding (BPE) with Llama pre-tokenization pattern optimized for Turkish morphological structure.

Training Details

Pre-training:

• Dataset: Turkish-dominant corpus totaling approximately 112.7 billion tokens
  • Legal Sources:
    • Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
    • Council of State (Danıştay): 151K sequences, ~0.11B tokens
    • Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
  • General Turkish Sources:
    • FineWeb2: General Turkish web data
    • CulturaX: Multilingual corpus (Turkish subset)
    • Total general Turkish: 212M sequences, ~96.17B tokens
  • Data Processing: SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
• Training Method: Masked Language Modeling (MLM) with 15% masking probability
• Masking Strategy: 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
• Framework: MosaicML Composer with Decoupled StableAdamW optimizer
• Learning Rate: 5×10⁻⁴ with warmup_stable_decay schedule
• Precision: BF16 mixed precision
• Hardware Infrastructure:
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Compute Nodes: 16 nodes
  • GPUs: 64× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
  • Node Configuration: Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
  • Interconnect: 800 Gb/s InfiniBand for distributed training
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
  • Distributed Training: Multi-node distributed training across 16 nodes with InfiniBand interconnect

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
• Training Framework: Sentence Transformers
• Optimization: Contrastive learning on Turkish passage ranking dataset
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
• Optimizer: AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Base-TR-Retrieval achieves strong performance with 55.86 MTEB Score and 47.52 Legal Score, demonstrating effectiveness for Turkish legal retrieval tasks.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.

Column abbreviations: MTEB = mean performance across task types; Legal = weighted average of Contracts, Regulation, Caselaw; Classification = accuracy on Turkish classification tasks; Clustering = V-measure on clustering tasks; Pair Classification = average precision on pair classification tasks like NLI; Retrieval = nDCG@10 on information retrieval tasks; Semantic Textual Similarity = Spearman correlation; Contracts = nDCG@10 on legal contract retrieval; Regulation = nDCG@10 on regulatory text retrieval; Caselaw = nDCG@10 on case law retrieval; Number of Parameters = number of model parameters; Model Type = model type (Embedding, CLM-Embedding, Masked Language Model). Bold values indicate the highest score in each column.

Key Findings:

• The model achieves strong performance on Turkish legal retrieval tasks with 55.86 MTEB Score and 47.52 Legal Score
• Strong performance on Contracts retrieval (80.40 nDCG@10) demonstrates effectiveness for legal document search
• Post-training on MS MARCO-TR significantly improves retrieval capabilities compared to base MLM models

Post-Training Performance Analysis

The following visualization shows the impact of post-training on retrieval performance:

Post-Training Retrieval Performance Comparison. Post-trained models (Mursit-Base-TR-Retrieval and Mursit-Large-TR-Retrieval) show significant improvements in legal domain retrieval tasks compared to base MLM models.

Reproducibility

To reproduce the benchmark results and training procedures for this model, please refer to:

• Post-Training: github.com/newmindai/mecellem-models/training/post-training-retrieval - Contains code and configurations for post-training retrieval models on MS MARCO-TR dataset.
• Embedding Benchmark Results: github.com/newmindai/mecellem-models/benchmark/embedding_model - Contains code and evaluation configurations for reproducing MTEB-Turkish benchmark results.

Usage

Installation

pip install sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("newmindai/Mursit-Base-TR-Retrieval")

# Encode sentences
sentences = [
    "Türk hukuk sistemi medeni hukuk geleneğine dayanır",
    "Anayasa Türkiye Cumhuriyeti'nin temel hukuk belgesidir",
    "Borçlar Kanunu sözleşmeleri düzenler"
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 768)

# Calculate similarity
similarities = model.similarity(embeddings, embeddings)
print(similarities)

Information Retrieval

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("newmindai/Mursit-Base-TR-Retrieval")

# Query and documents
query = "Sözleşme feshi nasıl yapılır?"
documents = [
    "Sözleşmeler yazılı olarak feshedilebilir.",
    "İş kanunu çalışma koşullarını düzenler.",
    "Fesih bildirimi noter aracılığıyla yapılmalıdır."
]

# Encode
query_embedding = model.encode(query, convert_to_tensor=True)
doc_embeddings = model.encode(documents, convert_to_tensor=True)

# Compute similarity scores
scores = util.cos_sim(query_embedding, doc_embeddings)[0]

# Rank documents
results = [(doc, score.item()) for doc, score in zip(documents, scores)]
results.sort(key=lambda x: x[1], reverse=True)

for doc, score in results:
    print(f"Score: {score:.4f} - {doc}")

Semantic Search Example

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("newmindai/Mursit-Base-TR-Retrieval")

# Legal document corpus
corpus = [
    "İş Kanunu'na göre işçinin haklı fesih sebepleri arasında ücretin ödenmemesi yer alır.",
    "Kira sözleşmesinin süresi dolduğunda taraflar yenileme yapabilir.",
    "Ticaret hukukunda anonim şirketlerin kuruluş sermayesi en az 50.000 TL olmalıdır."
]

# Query
query = "İşçi hangi durumlarda iş sözleşmesini feshedebilir?"

# Encode
query_embedding = model.encode(query)
corpus_embeddings = model.encode(corpus)

# Find most similar documents
hits = util.semantic_search(query_embedding, corpus_embeddings, top_k=3)
print(f"Query: {query}")
for hit in hits[0]:
    print(f"Score: {hit['score']:.4f} - {corpus[hit['corpus_id']]}")

ONNX Model Inference

This script demonstrates how to use the ONNX model from Hugging Face for text embedding generation.

Exporting Model to ONNX

To export the model to ONNX format, use the optimum-cli command:

optimum-cli export onnx \
  -m newmindai/Mursit-Base-TR-Retrieval \
  --task feature-extraction \
  onnx/MursitBaseTRRetrieval

This will create the model.onnx file in the specified output directory.

Installation

pip install onnxruntime-gpu transformers huggingface_hub numpy

Usage

import onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
import numpy as np

model_id = "newmindai/Mursit-Base-TR-Retrieval"

# Load tokenizer and download ONNX model from Hugging Face
tokenizer = AutoTokenizer.from_pretrained(model_id)
onnx_path = hf_hub_download(repo_id=model_id, filename="model.onnx")

# Use GPU if available, otherwise fallback to CPU
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"] if "CUDAExecutionProvider" in ort.get_available_providers() else ["CPUExecutionProvider"]

sess = ort.InferenceSession(onnx_path, providers=providers)

texts = ["This is a test"]
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="np")

outputs = sess.run(None, {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64),
})

embeddings = outputs[-1]  # sentence_embedding is usually the last output
print(embeddings.shape)
print(embeddings[:1])

Features

• Automatic GPU/CPU selection: Uses CUDA if available, otherwise falls back to CPU
• Hugging Face integration: Downloads model files directly from Hugging Face Hub
• Simple API: Easy-to-use interface for text embedding generation

Use Cases

• Semantic search in Turkish legal documents
• Legal document retrieval and ranking
• Contract similarity and matching
• Regulation compliance checking
• Case law research and discovery
• Question answering systems for legal domain
• Cross-lingual information retrieval (Turkish-English)
• Duplicate detection in legal texts
• Text clustering and classification

Reproducibility

To reproduce the MLM benchmark results for this model, please refer to:

• MLM Benchmark Results: github.com/newmindai/mecellem-models/benchmark/mlm - Contains code and evaluation configurations for reproducing MLM accuracy scores on Turkish datasets using the 80-10-10 masking strategy.

Acknowledgments

This work was supported by the EuroHPC Joint Undertaking through project etur46 with access to the MareNostrum 5 supercomputer, hosted by Barcelona Supercomputing Center (BSC), Spain. MareNostrum 5 is owned by EuroHPC JU and operated by BSC. We are grateful to the BSC support team for their assistance with job scheduling, environment configuration, and technical guidance throughout the project.

The numerical calculations reported in this work were fully/partially performed at TÜBİTAK ULAKBİM, High Performance and Grid Computing Center (TRUBA resources). The authors gratefully acknowledge the know-how provided by the MINERVA Support for expert guidance and collaboration opportunities in HPC-AI integration.

References

If you use this model, please cite our paper:

@article{mecellem2026,
  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
  author={Uğur, Özgür and Göksu, Mahmut and Çimen, Mahmut and Yılmaz, Musa and Şavirdi, Esra and Demir, Alp Talha and Güllüce, Rumeysa and Çetin, İclal and Sağbaş, Ömer Can},
  journal={arXiv preprint arXiv:2601.16018},
  year={2026},
  month={January},
  url={https://arxiv.org/abs/2601.16018},
  doi={10.48550/arXiv.2601.16018},
  eprint={2601.16018},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Base Model References

@inproceedings{modernbert2025,
  title={ModernBERT: A Modern Bidirectional Encoder Transformer},
  author={Answer.AI and LightOn},
  booktitle={Proceedings of the 2025 Conference on Language Models},
  year={2025}
}

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation}, 
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}