SPARSEUP
TL;DR: SPARSEUP is a strong and efficient sparse retriever trained on the same backbone family and fine-tuning data as DenseOn / LateOn (149M, ModernBERT). Strongest public vocabulary-based sparse encoder we know of under 150M on BEIR-13 (56.4 nDCG@10). Apache 2.0. Use it!
For the full story, see the blog post.
About
LightOn released open data and a recipe to train DenseOn (single-vector) and LateOn (late interaction). SPARSEUP fills the missing slot: a learned sparse embedding model, so the three architectures can be compared with backbone and data held fixed. The model is initialized from LateOn-unsupervised with ModernBERTβs MLM head / decoder.bias grafted on (decoder.weight stays tied to LateOn embeddings). Contrastive fine-tuning on LightOnβs mixture (7 hard negatives sampled from 50, plus in-batch). No cross-encoder distillation.
Three extra knobs vs vanilla SPLADE-like model:
logit_shift=15βlog(1 + ReLU(x β 15))so ReLU support starts sparseposition_top_k=12β each token keeps 12 vocab dims max before pooling (expansion budget)vocab_fold=case_spaceβ fold case /Δsurface forms onto one id after pooling (avoid duplication of surface forms = improve efficiency). Voc goes from 50k to ~34k.
Prefixes [Q] / [D] are attended but not pooled. Scoring is dot product.
BEIR (nDCG@10)
Controlled comparison against LateOn and DenseOn (same backbone / data). SPARSEUP uses Seismic (approximate); their numbers are exact search.
Model Avg Avg (w/o MS MARCO) LateOn 57.9 58.9 DenseOn 56.9 57.9 SPARSEUP 55.4 56.4 Comparison against competitive sparse encoders. Best model at <150M parameters.
Model Avg (wo/ MS MARCO) splade-v3 51.7 granite-embedding-30m-sparse 50.6 ModernBERT-VT 52.4 opensearch-neural-sparse-encoding-v1 52.4 opensearch-neural-sparse-encoding-doc-v3-gte 54.6 SPARSEUP 56.4 LACONIC-1B** 58.7
SPARSEUP is fast
Using dedicated infra (Seismic), it can reach sub-ms latency on MS MARCO (single-threaded).
Model description
- Model Type: Sentence Transformer or bare Transformers (see below)
- Encoder Type (sparse): MLM β
log1p(ReLU(x β ΞΈ))β per-position top-k β max-pool β vocab fold - Output Dimensionality: ~34k (sparse)
- Base: ModernBERT-base (149M), from LateOn-unsupervised + grafted MLM head
- Language: English
- Similarity: dot
- Query / doc prefixes:
[Q]/[D] - Max lengths (eval): query 128, document 512
- License: Apache 2.0
Usage
Transformers
from transformers import AutoModel
model = AutoModel.from_pretrained("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
Encoding
encode_to_dict(texts, kind="document", top_k=None) β encodes texts and returns one {token: weight} dict per text, sorted by descending weight. kind ("query"/"document") picks the prefix and max length; top_k caps how many terms you get back (None = all non-zero).
queries = ["England football player highest paid", "NYC capital which country?"]
model.encode_to_dict(queries, kind="query", round_to=2, top_k=10)
[{'Δ England': 2.96,
'Δ football': 2.68,
'Δ paid': 2.65,
'Δ player': 2.57,
'Δ highest': 2.52,
'Δ players': 2.19,
'Δ pay': 2.04,
'Δ english': 2.03,
'Δ eng': 2.03,
'Δ footballer': 1.97},
{'Δ capital': 3.25,
'Δ NYC': 3.12,
'Δ cap': 3.12,
'Δ country': 3.12,
'Δ ny': 3.09,
'Δ York': 2.87,
'Δ countries': 2.59,
'Δ capit': 2.55,
'Δ nation': 2.47,
'Δ Capitol': 2.45}]
Scoring
documents = [
"Harry Kane is the England captain and among the highest-paid footballers.",
"New York City is in the United States, not a national capital.",
]
q = model.encode(queries, kind="query")
d = model.encode(documents, kind="document")
print(model.score(q, d))
#tensor([[113.2993, 0.0000],
# [ 7.0030, 99.1290]])
Diagnosis
We also provide utilities to better visually inspect the content of predictions.
- prediction with attributions:
render(texts, kind="document", top_k=25, width=36, color=None)β returns a terminal bar chart string: one block per text, one line per expansion term with a heat-colored bar proportional to weight, plus the input subtoken that produced it (<- source@pos, for pure expansions).widthis the bar length in characters;color=Noneauto-detects a TTY, so passcolor=Truein Jupyter.
print(model.render(queries, kind="query", top_k=10))
query Β· England football player highest paid
England ββββββββββββββββββββββββββββββββββββ 2.96 <- Β·England@2
football βββββββββββββββββββββββββββββββββ 2.68 <- football@3
paid ββββββββββββββββββββββββββββββββ 2.65 <- paid@6
player βββββββββββββββββββββββββββββββ 2.57 <- player@4
highest βββββββββββββββββββββββββββββββ 2.52 <- highest@5
players βββββββββββββββββββββββββββ 2.19 <- player@4 <exp>
pay βββββββββββββββββββββββββ 2.04 <- paid@6 <exp>
english βββββββββββββββββββββββββ 2.03 <- Β·England@2 <exp>
eng βββββββββββββββββββββββββ 2.03 <- Β·England@2 <exp>
footballer ββββββββββββββββββββββββ 1.97 <- player@4 <exp>
query Β· NYC capital which country?
capital ββββββββββββββββββββββββββββββββββββ 3.25 <- capital@4
NYC βββββββββββββββββββββββββββββββββββ 3.12 <- Β·C@3 <exp>
cap βββββββββββββββββββββββββββββββββββ 3.12 <- capital@4 <exp>
country βββββββββββββββββββββββββββββββββββ 3.12 <- country@6
ny ββββββββββββββββββββββββββββββββββ 3.09 <- Β·NY@2
York ββββββββββββββββββββββββββββββββ 2.87 <- Β·NY@2 <exp>
countries βββββββββββββββββββββββββββββ 2.59 <- country@6 <exp>
capit ββββββββββββββββββββββββββββ 2.55 <- capital@4 <exp>
nation βββββββββββββββββββββββββββ 2.47 <- country@6 <exp>
Capitol βββββββββββββββββββββββββββ 2.45 <- capital@4 <exp>
- visual attribution:
highlight(texts, kind="document", reduce="sum", color=None)β returns the original texts with their firing words lit up, intensity being each word's share of the vector's total mass (reduce="sum", default) or its single strongest term (reduce="max"). On a TTY it uses reverse-video heat colors; in plain mode it wraps words in tiered markers β¦strongβ§ Β«midΒ» βΉweakβΊ (samecolor=Truecaveat for Jupyter).
document = ["The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated."]
print(model.highlight(document, kind="document", color=True))
# type: use color=True for a more visual rendering. β¦ β§ / Β« Β» / βΉ βΊ represent strong / mid / weak attributions.
βΉTheβΊ β¦presenceβ§ Β«ofΒ» β¦communicationβ§ β¦amidβ§ β¦scientificβ§ β¦mindsβ§ Β«wasΒ» β¦equallyβ§ β¦importantβ§ Β«toΒ» the β¦successβ§ Β«ofΒ» Β«theΒ» β¦Manhattanβ§ β¦Projectβ§ Β«asΒ» βΉscientificβΊ β¦intellectβ§ wasΒ«.Β» βΉTheβΊ Β«onlyΒ» β¦cloudβ§ β¦hangingβ§ Β«overΒ» the β¦impressiveβ§ β¦achievementβ§ of the β¦atomicβ§ β¦researchersβ§ βΉandβΊ β¦engineersβ§ βΉisβΊ what their Β«successΒ» Β«trulyΒ» β¦meant;β§ β¦hundredsβ§ Β«ofΒ» β¦thousandsβ§ Β«ofΒ» β¦innocentβ§ β¦livesβ§ β¦obliteratedβ§.
Sentence Transformers
The model can be used with Sentence Transformers. First install the lib:
pip install -U sentence-transformers
from sentence_transformers import SparseEncoder
model = SparseEncoder("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
queries = [
"England football player highest paid",
"NYC capital which country?",
]
q = model.encode_query(queries)
print(model.decode(q, top_k=10))
# [[('Δ England', 2.96), ('Δ football', 2.68), ...],
# [('Δ capital', 3.25), ('Δ NYC', 3.12), ...]]
documents = [
"Harry Kane is the England captain and among the highest-paid footballers.",
"New York City is in the United States, not a national capital.",
]
d = model.encode_document(documents)
print(model.similarity(q, d))
# tensor([[113.29, 0.00],
# [ 7.00, 99.13]])