SlideOn-SAE

English Sparse Autoencoder by LightOn

🎯 TL;DR: The pretrained sparse autoencoder used to initialize SlideOn. It learns 32,768 latent features from frozen DenseOn token representations and provides a starting point for supervised sparse retrieval fine-tuning. Use the final SlideOn checkpoint for the retrieval results reported in our blog.

Model Details

SlideOn-SAE maps L2-normalized DenseOn token representations to their top-32 latent activations, followed by ReLU. Its decoder reconstructs the original token representations during SAE pretraining. For retrieval, the included adapter max-pools token activations into one sparse vector per query or document.

Property	Value
Backbone	DenseOn / ModernBERT-base
Training stage	SAE pretraining, before supervised retrieval fine-tuning
Token representation	768 dimensions, L2-normalized before sparse encoding
Latent dimensionality	32,768 learned features
Token sparsity	Top-32 latent selection followed by ReLU
Sequence pooling	Maximum over token activations
Similarity function	Dot product
Language	English

The checkpoint includes the DenseOn backbone, SAE encoder, and original SAE decoder in a single model.safetensors file. The model and tokenizer load directly from this repository.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning. Training code and the recorded training configuration are available in the code repository.

Usage

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"

Load the SAE initialization and encode text into sparse vectors:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/SlideOn-SAE", trust_remote_code=True)

query_embeddings = model.encode_query(["What is sparse retrieval?"])
document_embeddings = model.encode_document([
    "Sparse retrieval uses inverted indexes to search sparse vectors."
])

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 32768]) torch.Size([1, 32768])

The model loaded above can also be fine-tuned with Sentence Transformers' SparseEncoderTrainer. Our code repository provides the custom retrieval loss and training scripts used for our fine-tuning recipe.

Use max_active_dims=128 to keep the 128 latents with the highest weights in each pooled vector and set the rest to zero. Omit it or use None for the saved unpruned defaults; 0 is invalid. This is independent of token_k (32 by default).

Related Checkpoints

Model	Description
SlideOn-SAE (this model)	DenseOn-based SAE initialization
SlideOn	English sparse retriever after supervised fine-tuning
mSlideOn-SAE	mLateOn-based multilingual SAE initialization
mSlideOn	Multilingual sparse retriever after supervised fine-tuning

Citation

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}