📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation

mSlideOn

Multilingual Sparse Retrieval Model by LightOn

SlideOn | mSlideOn | SlideOn-SAE | mSlideOn-SAE

🎯 TL;DR: A 339M-parameter multilingual sparse retrieval model built from mLateOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. mSlideOn improves on mDenseOn by 4.76 points on MIRACL and 10.19 points on MLDR, and reaches 74.78 on MTEB Code, surpassing both mDenseOn and mLateOn. It brings multilingual, long-context, and code retrieval to sparse representations compatible with inverted-index search, with Matryoshka training for adjustable query and document sparsity.

About the SlideOn / mSlideOn Family

With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our blog post for the full method, results, and ablations.

mSlideOn

mSlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. Its sparse encoder lifts mLateOn's normalized 128-dimensional token representations to 768 dimensions and applies a residual MLP before selecting the top-32 latent activations and applying ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

mSlideOn notably:

• Reaches 56.00 on BEIR-15, the highest score among the multilingual sparse models in our comparison.
• Reaches 63.21 on MIRACL target languages and 62.78 across all 18 languages, improving on mDenseOn by 4.76 points on the full benchmark.
• Reaches 75.79 on MLDR target languages and 61.68 across all 13 languages, outperforming every dense baseline in our comparison.
• Reaches 74.78 on MTEB Code, ahead of mDenseOn (71.53) and mLateOn (73.48).
• Supports inputs of up to 8,192 tokens and independently adjustable query and document sparsity.

Results

Headline Results (nDCG@10)

Scores are average nDCG@10 on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction comparisons follow our mDenseOn and mLateOn blog post.

MIRACL tgt covers Arabic, English, French, German, and Spanish; MLDR tgt covers Arabic, English, French, German, Italian, Portuguese, and Spanish. These are benchmark languages included in retrieval training. Full MIRACL covers 18 languages, full MLDR covers 13, and MTEB Code averages 12 retrieval tasks.

We evaluate MIRACL using the dev split of mteb/miracl-hard-negatives and MLDR using the test split of mteb/MultiLongDocRetrieval.

Retrieval results on BEIR-15, MIRACL, MLDR, and MTEB Code (↑ higher is better), with retrieval FLOPS (↓ lower is better). Blue denotes our models; bold values mark the best result within each family for each metric. OpenSearch multilingual corresponds to opensearch-neural-sparse-encoding-multilingual-v1.

Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.

Key Findings

Multilingual and long-document retrieval. mSlideOn improves over mDenseOn on full MIRACL and MLDR by 4.76 and 10.19 points, respectively, while remaining behind mLateOn. The per-language results show gains beyond the languages included in retrieval training.

Code retrieval. mSlideOn outperforms every sparse and late-interaction baseline in this comparison on MTEB Code.

English retrieval. mSlideOn reaches 56.00 on BEIR-15, compared with 56.70 for mDenseOn and 57.56 for mLateOn. The gains on multilingual, long-document, and code tasks therefore come with a modest reduction in English benchmark performance.

Per-dataset and per-language differences in nDCG@10 points for mSlideOn versus mDenseOn. Teal bars indicate gains and red bars indicate declines relative to mDenseOn. Blue labels mark languages included in retrieval training.

Related Checkpoints

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning.

Model Details

Model Description

Usage

Direct Usage (Sentence Transformers)

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"

Load the model and encode multilingual queries and documents:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/mSlideOn", trust_remote_code=True)

queries = [
    "Qu'est-ce qui provoque une éclipse solaire ?",
    "Was verursacht eine Sonnenfinsternis?",
]
documents = [
    "A solar eclipse occurs when the Moon passes between the Sun and Earth.",
    "Um eclipse solar ocorre quando a Lua passa entre o Sol e a Terra.",
    "Wind turbines convert the kinetic energy of wind into electricity.",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([3, 32768])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

encode_query() inserts the [Q] query marker, and encode_document() inserts the [D] document marker, immediately after the beginning-of-sequence token. Pass raw text; the model adds these special tokens automatically and does not use query: or document: text prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.

Choosing Query and Document Budgets

Set max_active_dims=N to keep the top-N latents with the highest weights in each pooled vector and set the rest to zero. You can choose a different N for queries and documents. This is separate from the per-token top-32 operation and does not change the vector dimensionality. For example, to use q-128 / d-1024:

query_embeddings = model.encode_query(queries, max_active_dims=128)
document_embeddings = model.encode_document(documents, max_active_dims=1024)

Omitting max_active_dims, or setting it to None, uses unpruned vectors, matching the headline results.

Pruning budgets must be positive integers; 0 is invalid.

Dimensions represent learned latents, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; SparseEncoder.decode() cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.

Citation

If you find this work useful for your research, please consider citing us:

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}