ALIA MrBERT Spanish Legal and Administrative Reranker Model

This repository contains ALIA MrBERT Spanish Legal and Administrative Reranker, a Spanish legal domain cross-encoder (reranker) model for information retrieval and document ranking tasks. It is built upon MrBERT-es, a bilingual (Spanish–English) foundational language model based on the ModernBERT architecture, and fine-tuned on domain-specific legal and administrative data using a Curriculum Learning strategy.

[!WARNING] DISCLAIMER: This model is a domain-specific proof-of-concept designed to demonstrate retrieval capabilities in the Spanish legal and administrative domain. While optimized for this domain, results should be verified against official legal sources. The model may fail in out-of-domain or adversarial inputs.

---

Model Details

Model Lineage

ModernBERT (architecture)
       ↓
  MrBERT-es (BSC-LT)
  Bilingual ES/EN encoder
  150M parameters
       ↓
  ALIA-MrBERT-es-legal-administrative-reranker (SINAI)
  Legal domain fine-tuning
  Curriculum Learning + Hard Negatives

Key Features

• 🔍 Domain: Spanish legal and administrative texts
• 📐 Architecture: ModernBERT Cross-Encoder (reranker)
• 📏 Context length: up to 8,192 tokens
• 🎓 Training strategy: Curriculum Learning (easy → medium → hard)
• ⚙️ Negative mining: Positive-Aware Hard Negative Mining

Architecture

This model uses the same base architecture as MrBERT-es, formatted as a Cross-Encoder for sentence pair classification:

---

Training

Training Strategy: Curriculum Learning

The model was fine-tuned using a Curriculum Learning strategy, progressively increasing the difficulty of training examples. For this Cross-Encoder, the training focused on the most challenging examples: pairs that a Bi-Encoder might struggle to distinguish.

The dataset consists of text pairs incorporating hard negatives mined from the corpus SINAI/ALIA-es-legal-administrative-triplets/train. For the Cross-Encoder, the data is flattened into independent {query, document, label} pairs, where label is either 1.0 (relevant) or 0.0 (irrelevant).

Refinement with Hard Negatives: Training uses mined hard negatives to force the model to distinguish fine-grained nuances. Examples are processed in order of increasing difficulty (easy → medium → hard) to improve convergence and generalization.

Hyperparameter Optimization

Before training, hyperparameter search was conducted using Optuna (20 trials) to maximize NDCG@10 (with fallback to MRR@10) on a control subset:

• Sampler: TPESampler (Tree-structured Parzen Estimator)
• Pruner: MedianPruner with an OptunaPruningCallback reporting NDCG@10
• Evaluator: CrossEncoderRerankingEvaluator

Final Training Hyperparameters

Training Framework

---

Intended Use

Direct Use

This model is designed for document reranking and semantic matching tasks in the Spanish legal and administrative domain. Primary use cases include:

• RAG pipelines: Reranking retrieved context chunks for language models
• Search pipelines: Improving initial retrieval (e.g. BM25 or Bi-encoder) by doing precise cross-encoding over the top-k results
• Legal text matching: Determining high-resolution entailment or relevance between queries and legal clauses

Out-of-Scope Use

• General-domain retrieval (the model is specialized for legal/administrative Spanish)
• Fast, large-scale search across millions of documents (use a Bi-encoder first, then rerank the top-k results with this Cross-encoder)
• Cross-lingual retrieval beyond Spanish

---

How to Use

With sentence-transformers

from sentence_transformers import CrossEncoder

model = CrossEncoder("SINAI/ALIA-MrBERT-es-legal-administrative-reranker")

query = "¿Cuáles son los requisitos para solicitar una prestación por desempleo?"
documents = [
    "El trabajador que cese en su actividad laboral tendrá derecho a la prestación por desempleo...",
    "La prestación por desempleo contributiva se reconoce a quienes hayan cotizado al menos 360 días...",
    "El subsidio de incapacidad temporal requiere un certificado médico en vigor.",
]

# We want to score the query with each document
pairs = [[query, doc] for doc in documents]

scores = model.predict(pairs)
print(scores)

---

Evaluation

The model was evaluated using the MTEB (Massive Text Embedding Benchmark) framework, adapted for the legal domain. The main reported metric is NDCG@10 (Normalized Discounted Cumulative Gain at k=10), which is the standard metric used in retrieval leaderboards and aligns with the metric reported in the MrBERT family.

Evaluation Datasets

Results

The following table reports the performance of the model compared to leading commercial/open-weight generalist models:

Note: The BGE reranker and Nemotron reranker are significantly larger scale models (billions of parameters), yet our domain-specific 150M parameter cross-encoder performs comparably or better on specialized legal Spanish datasets (e.g., ssld, pairs650, pairs1.3k).

---

Limitations and Biases

Known Limitations

• Domain specificity: The model is highly optimized for Spanish legal texts. Its zero-shot capabilities on general domains are weaker compared to massive generalist rerankers.
• Latency: Being a cross-encoder, it can be computationally heavy to score many document-query pairs. Use it to rerank a maximum of typical 20-100 top documents fetched by a fast bi-encoder.
• Legal accuracy: Semantic similarity does not guarantee legal correctness.

Biases

• The model reflects biases present in Spanish legislation, case law, and administrative corpora.

---

Additional Information

License

Apache License, Version 2.0

Citation

If you use this model in your research, please cite:

@misc{ALIA-MrBERT-es-legal-administrative-reranker,
  title        = {ALIA MrBERT Spanish Legal and Administrative Reranker Model},
  author       = {SINAI Research Group, Universidad de Jaén},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-legal-administrative-reranker}}
}

Please also cite the base model:

@misc{tamayo2026mrbertmodernmultilingualencoders,
      title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation}, 
      author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2026},
      eprint={2602.21379},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.21379}, 
}

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.

Acknowledgments

This dataset has been generated thanks to CEATIC (Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – UJA (Universidad de Jaén) which provided the needed computational resources on its clusters.

---

Contact: ALIA Project - SINAI Research Group - Universidad de Jaén

More Information: SINAI Research Group | ALIA-UJA Project