mDenseOn-spa-32768

Spanish vocabulary-trimmed version of lightonai/mDenseOn.

How this model was built

This model was created by trimming the vocabulary of lightonai/mDenseOn for use with Spanish text.

Inspiration

This project was inspired by Introduction to Trimming by Loïck Bourdois and collaborators, which explores vocabulary trimming as a way to reduce the size of multilingual models. Their work provided the motivation for this Spanish adaptation and its vocabulary selection approach. The implementation and evaluation results presented here correspond to this project.

Procedure

1. Token frequencies were computed using the Spanish (spa) split of lbourdois/fineweb-2-trimming.
2. Special tokens and the tokens required by the tokenizer's byte representation mechanisms were preserved.
3. Tokens were selected by frequency, including their required BPE dependencies, within the target vocabulary budget.
4. The vocabulary and BPE merges were rebuilt, assigning contiguous token IDs.
5. The original embeddings of the retained tokens were copied to their new positions.
6. Non-special added tokens are retained only when included in the selected vocabulary.

The resulting vocabulary contains 32,768 tokens. No additional training was performed. All other model weights remain unchanged.

Model reduction

The following measurements and retrieval results correspond to the evaluated mDenseOn-spa-32768 build of lightonai/mDenseOn, without additional training. Re-run evaluation after changing token selection or model weights.

Disk size refers to the FP32 model.safetensors files, with 1 MB = 1,000,000 bytes. Parameter and vocabulary reductions are exact; sizes and percentages are rounded. The Transformer backbone weights are preserved, and retained token embeddings are copied from the original.

Retrieval evaluation

Both models were evaluated on identical queries and candidate documents, using the model's query and document prompts, normalized embeddings, and cosine similarity. The maximum sequence length was 1,024 tokens and the batch size was 4. Inference ran on an NVIDIA GeForce RTX 3060 Laptop GPU. All benchmark inputs were read using streaming and held in RAM. Scores below use a 0-1 scale. Delta is trimmed minus original.

MIRACL-es: reduced shared corpus

All 648 Spanish development queries were evaluated against a shared corpus of 6,410 unique passages, formed from the union of their annotated positive and negative passages. Every query searched the same corpus.

This is a reduced-corpus comparison, not the official full-corpus MIRACL benchmark. Its scores are not comparable to full-corpus leaderboard results.

The tokenizer exceeded the sequence limit for 2 documents with the original model and 3 with the trimmed model; those documents were truncated. No queries were truncated.

Dataset: MIRACL.

XQuADRetrieval-es: complete Spanish task

All 1,184 Spanish validation queries were evaluated against all 240 passages in the Spanish corpus of the published XQuADRetrieval task. Corpus, queries, and relevance judgments were loaded using streaming. No queries or documents were truncated for either model.

Dataset: XQuADRetrieval.

MuPLeR-retrieval-es: complete Spanish legal task

All 200 Spanish test queries were evaluated against all 10,000 legal passages in the Spanish MuPLeR corpus. The passages come from European Union legislation (DGT-Acquis); the queries are synthetic. Every query searched the complete Spanish corpus. No queries or documents were truncated for either model.

Dataset: MuPLeR-retrieval.

Interpretation

Trimming reduces weight-file size by 55.85%, but nDCG@10 is lower on all three evaluated tasks. On a 0-100 scale, nDCG@10 decreases by 5.37 points on the reduced MIRACL corpus, 1.34 points on XQuADRetrieval, and 2.19 points on MuPLeR. Recall@10 remains unchanged on MuPLeR, while MRR@10 decreases by 2.90 points. These results describe the three evaluation settings above; they do not establish equivalent quality across all Spanish retrieval tasks.