ALIA MrBERT Spanish Legal and Administrative Embeddings Model

This repository contains ALIA MrBERT Spanish Legal and Administrative Embeddings, a Spanish legal domain bi-encoder model for semantic similarity and information retrieval tasks. It is built upon MrBERT-es, a bilingual (Spanish–English) foundational language model based on the ModernBERT architecture, and fine-tuned on domain-specific legal and administrative data using a Curriculum Learning strategy.

[!WARNING] DISCLAIMER: This model is a domain-specific proof-of-concept designed to demonstrate retrieval capabilities in the Spanish legal and administrative domain. While optimized for this domain, results should be verified against official legal sources. The model may fail in out-of-domain or adversarial inputs.

---

Model Details

Model Lineage

ModernBERT (architecture)
       ↓
  MrBERT-es (BSC-LT)
  Bilingual ES/EN encoder
  150M parameters
       ↓
  ALIA-MrBERT-es-legal-administrative-embeddings (SINAI)
  Legal domain fine-tuning
  Curriculum Learning + Hard Negatives

Key Features

• 🔍 Domain: Spanish legal and administrative texts
• 📐 Architecture: ModernBERT with Mean Pooling (bi-encoder)
• 📏 Long context: up to 8,192 tokens
• 🎓 Training strategy: Curriculum Learning (easy → medium → hard)
• ⚙️ Negative mining: Positive-Aware Hard Negative Mining (NVIDIA approach)

Architecture

This model uses the same base architecture as MrBERT-es, extended with a Mean Pooling layer for sentence-level embeddings:

---

Training

Training Strategy: Curriculum Learning

The model was fine-tuned using a two-phase Curriculum Learning strategy and progressively increasing the difficulty of training examples thanks to SINAI/ALIA-es-legal-administrative-triplets/train:

Phase 1 – Contrastive Learning with Random Negatives: Training uses {query, relevant_doc, [irrelevant_docs]} with in-batch negatives. Examples are sorted by difficulty across 3 sub-phases (2 epochs each).

Phase 2 – Advanced Refinement with Hard Negatives: Refinement using mined hard negatives with Positive-Aware Mining (NVIDIA approach) to avoid false negatives. A candidate is only considered a negative if:

score < score_positive - margin   (margin = 0.05)

Hyperparameter Optimization

Before training, hyperparameter search was conducted using Optuna (20 trials, subsets of 5,000 examples):

• Sampler: TPESampler (Tree-structured Parzen Estimator)
• Pruner: MedianPruner
• Storage: SQLite for trial persistence

Final Training Hyperparameters

Loss Function

• CachedMultipleNegativesRankingLoss: Enables training with large batches (256) without VRAM overflow, by recalculating embeddings in smaller sub-batches (cache size: 4).

Training Framework

---

Intended Use

Direct Use

This model is designed for semantic similarity and information retrieval tasks in the Spanish legal and administrative domain. Primary use cases include:

• Semantic search: Finding relevant legal documents from a query
• RAG pipelines: Generating context-enriched legal answers using retrieval-augmented generation
• Legal document clustering: Grouping similar legal texts by semantic content
• Duplicate detection: Identifying semantically similar legal clauses or articles

Out-of-Scope Use

• General-domain retrieval (model is specialized for legal/administrative Spanish)
• Cross-lingual retrieval beyond Spanish
• Use as a generative model (this is an encoder-only model)
• Legal advice or binding interpretations of legal texts

---

How to Use

With sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("SINAI/ALIA-MrBERT-es-legal-administrative-embeddings")

queries = ["¿Cuáles son los requisitos para solicitar una prestación por desempleo?"]
documents = [
    "El trabajador que cese en su actividad laboral tendrá derecho a la prestación por desempleo...",
    "La prestación por desempleo contributiva se reconoce a quienes hayan cotizado al menos 360 días...",
]

query_embeddings = model.encode(queries, prompt_name="query")
doc_embeddings = model.encode(documents)

scores = model.similarity(query_embeddings, doc_embeddings)
print(scores)

With transformers (manual)

import torch
from transformers import AutoTokenizer, AutoModel

model_name = "SINAI/ALIA-MrBERT-es-legal-administrative-embeddings"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["¿Cuáles son los plazos para interponer un recurso administrativo?"]
encoded = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
    outputs = model(**encoded)

embeddings = mean_pooling(outputs, encoded["attention_mask"])
print(embeddings.shape)  # (1, 768)

---

Evaluation

The model was evaluated using the MTEB (Massive Text Embedding Benchmark) framework, adapted for the legal domain. The main reported metric is NDCG@10 (Normalized Discounted Cumulative Gain at k=10), which is the standard metric used in retrieval leaderboards and aligns with the metric reported in the MrBERT family.

An additional evaluation was made thanks to ragas evaluation framework and MiniMaxAI/MiniMax-M2.5 language model. These metrics are calculated by averaging on each pair puntuation from particular subsets of some of the following datasets.

Evaluation Datasets

Results (NDCG@10)

BAAI/bge-m3 and Qwen/Qwen3-Embedding-0.6B are SOTA models with a size x4 larger than ours. Yet, our model shows comparable results.

Results (ragas)

Subset: triplets_queries1300_contexts1300 (triplets1.3k)

Subset: justicio_queries512_contexts154

Subset: ssld_queries500_contexts11234

Note: Each evaluation subset is named following the pattern {dataset}_queries{N}_contexts{M}, where N is the number of queries evaluated against M contexts taken from the datasets.

---

Limitations and Biases

Known Limitations

• Domain specificity: The model is optimized for Spanish legal and administrative texts. Performance may degrade significantly on general-domain or other specialized texts.
• Language: Although MrBERT-es supports Spanish and English, this fine-tuned model focuses on Spanish legal content.
• Legal accuracy: Semantic similarity does not guarantee legal correctness. Retrieved documents should always be verified by qualified professionals.
• Context length: Despite supporting up to 8,192 tokens, very long documents may require chunking strategies for optimal retrieval performance.

Biases

• The model may reflect biases present in the Spanish legal corpus used for training.
• It may underperform on legal texts from Latin American jurisdictions, as training focused on Spanish national legislation and administration.

---

Additional Information

License

Apache License, Version 2.0

Citation

If you use this model in your research, please cite:

@misc{ALIA-MrBERT-es-legal-administrative-embeddings,
  title        = {ALIA MrBERT Spanish Legal and Administrative Embeddings Model},
  author       = {SINAI Research Group, Universidad de Jaén},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-legal-administrative-embeddings}}
}

Please also cite the base model:

@misc{tamayo2026mrbertmodernmultilingualencoders,
      title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation}, 
      author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2026},
      eprint={2602.21379},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.21379}, 
}

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.

Acknowledgments

This dataset has been generated thanks to CEATIC ( Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – UJA (Universidad de Jaén) which provided the needed computational resources on its clusters.

---

Contact: ALIA Project - SINAI Research Group - Universidad de Jaén

More Information: SINAI Research Group | ALIA-UJA Project