mSlideOn-SAE

Multilingual Sparse Autoencoder by LightOn

🎯 TL;DR: The pretrained sparse autoencoder used to initialize mSlideOn. It learns 32,768 latent features from frozen mLateOn token representations and provides a starting point for supervised multilingual and code retrieval fine-tuning. Use the final mSlideOn checkpoint for the retrieval results reported in our blog.

Model Details

mSlideOn-SAE lifts L2-normalized, 128-dimensional mLateOn token representations to 768 dimensions and applies a residual MLP before projecting to 32,768 latent features, selecting the top-32 activations, and applying ReLU. Its linear decoder reconstructs the original 128-dimensional token representations during SAE pretraining. For retrieval, the included adapter max-pools token activations into one sparse vector per query or document.

Property	Value
Backbone	mLateOn / mmBERT-base
Training stage	SAE pretraining, before supervised retrieval fine-tuning
Token representation	128 dimensions, L2-normalized before sparse encoding
Sparse encoder	Lift to 768 dimensions, residual MLP, and latent projection
Latent dimensionality	32,768 learned features
Token sparsity	Top-32 latent selection followed by ReLU
Sequence pooling	Maximum over token activations
Similarity function	Dot product
Prompts	`[Q]` marker for queries; `[D]` marker for documents, inserted automatically after the beginning-of-sequence token
Languages	Multilingual and code

The checkpoint includes the mLateOn backbone and token projections, SAE encoder, and original SAE decoder in a single model.safetensors file. The model and tokenizer load directly from this repository.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning. Training code and the recorded training configuration are available in the code repository.

Usage

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"

Load the SAE initialization and encode text into sparse vectors:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/mSlideOn-SAE", trust_remote_code=True)

query_embeddings = model.encode_query([
    "What is sparse retrieval?",
    "Qu’est-ce que la recherche éparse ?",
])
document_embeddings = model.encode_document([
    "Sparse retrieval uses inverted indexes to search sparse vectors.",
    "La recherche éparse utilise des index inversés pour rechercher des vecteurs épars.",
])

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([2, 32768])

encode_query() and encode_document() insert the [Q] and [D] markers automatically. Pass raw text to both methods.

The model loaded above can also be fine-tuned with Sentence Transformers' SparseEncoderTrainer. Our code repository provides the custom retrieval loss and training scripts used for our fine-tuning recipe.

Use max_active_dims=128 to keep the 128 latents with the highest weights in each pooled vector and set the rest to zero. Omit it or use None for the saved unpruned defaults; 0 is invalid. This is independent of token_k (32 by default).

Related Checkpoints

Model	Description
mSlideOn-SAE (this model)	mLateOn-based multilingual SAE initialization
mSlideOn	Multilingual sparse retriever after supervised fine-tuning
SlideOn-SAE	DenseOn-based English SAE initialization
SlideOn	English sparse retriever after supervised fine-tuning

Citation

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}