SPARSEUP

TL;DR: SPARSEUP is a strong and efficient sparse retriever trained on the same backbone family and fine-tuning data as DenseOn / LateOn (149M, ModernBERT). Strongest public vocabulary-based sparse encoder we know of under 150M on BEIR-13 (56.4 nDCG@10). Apache 2.0. Use it!

For the full story, see the blog post.

About

LightOn released open data and a recipe to train DenseOn (single-vector) and LateOn (late interaction). SPARSEUP fills the missing slot: a learned sparse embedding model, so the three architectures can be compared with backbone and data held fixed. The model is initialized from LateOn-unsupervised with ModernBERT’s MLM head / decoder.bias grafted on (decoder.weight stays tied to LateOn embeddings). Contrastive fine-tuning on LightOn’s mixture (7 hard negatives sampled from 50, plus in-batch). No cross-encoder distillation.

Three extra knobs vs vanilla SPLADE-like model:

1. logit_shift=15 — log(1 + ReLU(x − 15)) so ReLU support starts sparse
2. position_top_k=12 — each token keeps 12 vocab dims max before pooling (expansion budget)
3. vocab_fold=case_space — fold case / Ġ surface forms onto one id after pooling (avoid duplication of surface forms = improve efficiency). Voc goes from 50k to ~34k.

Prefixes [Q] / [D] are attended but not pooled. Scoring is dot product.

BEIR (nDCG@10)

• Controlled comparison against LateOn and DenseOn (same backbone / data). SPARSEUP uses Seismic (approximate); their numbers are exact search.

  Model	Avg	Avg (w/o MS MARCO)
  LateOn	57.9	58.9
  DenseOn	56.9	57.9
  SPARSEUP	55.4	56.4

• Comparison against competitive sparse encoders. Best model at <150M parameters.

  Model	Avg (wo/ MS MARCO)
  splade-v3	51.7
  granite-embedding-30m-sparse	50.6
  ModernBERT-VT	52.4
  opensearch-neural-sparse-encoding-v1	52.4
  opensearch-neural-sparse-encoding-doc-v3-gte	54.6
  SPARSEUP	56.4
  LACONIC-1B**	58.7

SPARSEUP is fast

Using dedicated infra (Seismic), it can reach sub-ms latency on MS MARCO (single-threaded).

Model description

• Model Type: Sentence Transformer or bare Transformers (see below)
• Encoder Type (sparse): MLM → log1p(ReLU(x − θ)) → per-position top-k → max-pool → vocab fold
• Output Dimensionality: ~34k (sparse)
• Base: ModernBERT-base (149M), from LateOn-unsupervised + grafted MLM head
• Language: English
• Similarity: dot
• Query / doc prefixes: [Q] / [D]
• Max lengths (eval): query 128, document 512
• License: Apache 2.0

Usage

Transformers

from transformers import AutoModel
model = AutoModel.from_pretrained("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)

Encoding

encode_to_dict(texts, kind="document", top_k=None) — encodes texts and returns one {token: weight} dict per text, sorted by descending weight. kind ("query"/"document") picks the prefix and max length; top_k caps how many terms you get back (None = all non-zero).

queries = ["England football player highest paid", "NYC capital which country?"]
model.encode_to_dict(queries, kind="query", round_to=2, top_k=10)
[{'ĠEngland': 2.96,
  'Ġfootball': 2.68,
  'Ġpaid': 2.65,
  'Ġplayer': 2.57,
  'Ġhighest': 2.52,
  'Ġplayers': 2.19,
  'Ġpay': 2.04,
  'Ġenglish': 2.03,
  'Ġeng': 2.03,
  'Ġfootballer': 1.97},
 {'Ġcapital': 3.25,
  'ĠNYC': 3.12,
  'Ġcap': 3.12,
  'Ġcountry': 3.12,
  'Ġny': 3.09,
  'ĠYork': 2.87,
  'Ġcountries': 2.59,
  'Ġcapit': 2.55,
  'Ġnation': 2.47,
  'ĠCapitol': 2.45}]

Scoring

documents = [
    "Harry Kane is the England captain and among the highest-paid footballers.",
    "New York City is in the United States, not a national capital.",
]
q = model.encode(queries, kind="query")
d = model.encode(documents, kind="document")
print(model.score(q, d))
#tensor([[113.2993,   0.0000],
#        [  7.0030,  99.1290]])

Diagnosis

We also provide utilities to better visually inspect the content of predictions.

• prediction with attributions: render(texts, kind="document", top_k=25, width=36, color=None) — returns a terminal bar chart string: one block per text, one line per expansion term with a heat-colored bar proportional to weight, plus the input subtoken that produced it (<- source@pos, for pure expansions). width is the bar length in characters; color=None auto-detects a TTY, so pass color=True in Jupyter.

print(model.render(queries, kind="query", top_k=10))

query · England football player highest paid
   England █████████████████████████████████▓▒░   2.96  <- ·England@2
  football ██████████████████████████████▓▒░      2.68  <- football@3
      paid █████████████████████████████▓▒░       2.65  <- paid@6
    player ████████████████████████████▓▒░        2.57  <- player@4
   highest ████████████████████████████▓▒░        2.52  <- highest@5
   players ████████████████████████▓▒░            2.19  <- player@4 <exp>
       pay ██████████████████████▓▒░              2.04  <- paid@6 <exp>
   english ██████████████████████▓▒░              2.03  <- ·England@2 <exp>
       eng ██████████████████████▓▒░              2.03  <- ·England@2 <exp>
footballer █████████████████████▓▒░               1.97  <- player@4 <exp>

query · NYC capital which country?
  capital █████████████████████████████████▓▒░   3.25  <- capital@4
      NYC ████████████████████████████████▓▒░    3.12  <- ·C@3 <exp>
      cap ████████████████████████████████▓▒░    3.12  <- capital@4 <exp>
  country ████████████████████████████████▓▒░    3.12  <- country@6
       ny ███████████████████████████████▓▒░     3.09  <- ·NY@2
     York █████████████████████████████▓▒░       2.87  <- ·NY@2 <exp>
countries ██████████████████████████▓▒░          2.59  <- country@6 <exp>
    capit █████████████████████████▓▒░           2.55  <- capital@4 <exp>
   nation ████████████████████████▓▒░            2.47  <- country@6 <exp>
  Capitol ████████████████████████▓▒░            2.45  <- capital@4 <exp>

• visual attribution: highlight(texts, kind="document", reduce="sum", color=None) — returns the original texts with their firing words lit up, intensity being each word's share of the vector's total mass (reduce="sum", default) or its single strongest term (reduce="max"). On a TTY it uses reverse-video heat colors; in plain mode it wraps words in tiered markers ⟦strong⟧ «mid» ‹weak› (same color=True caveat for Jupyter).

document = ["The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated."]
print(model.highlight(document, kind="document", color=True))
# type: use color=True for a more visual rendering. ⟦ ⟧ / « » / ‹ › represent strong / mid / weak attributions.

‹The› ⟦presence⟧ «of» ⟦communication⟧ ⟦amid⟧ ⟦scientific⟧ ⟦minds⟧ «was» ⟦equally⟧ ⟦important⟧ «to» the ⟦success⟧ «of» «the» ⟦Manhattan⟧ ⟦Project⟧ «as» ‹scientific› ⟦intellect⟧ was«.» ‹The› «only» ⟦cloud⟧ ⟦hanging⟧ «over» the ⟦impressive⟧ ⟦achievement⟧ of the ⟦atomic⟧ ⟦researchers⟧ ‹and› ⟦engineers⟧ ‹is› what their «success» «truly» ⟦meant;⟧ ⟦hundreds⟧ «of» ⟦thousands⟧ «of» ⟦innocent⟧ ⟦lives⟧ ⟦obliterated⟧.

Sentence Transformers

The model can be used with Sentence Transformers. First install the lib:

pip install -U sentence-transformers

from sentence_transformers import SparseEncoder
model = SparseEncoder("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
queries = [
    "England football player highest paid",
    "NYC capital which country?",
]
q = model.encode_query(queries)
print(model.decode(q, top_k=10))
# [[('ĠEngland', 2.96), ('Ġfootball', 2.68), ...],
#  [('Ġcapital', 3.25), ('ĠNYC', 3.12), ...]]

documents = [
    "Harry Kane is the England captain and among the highest-paid footballers.",
    "New York City is in the United States, not a national capital.",
]
d = model.encode_document(documents)
print(model.similarity(q, d))
# tensor([[113.29,   0.00],
#         [  7.00,  99.13]])