LexSPLADE (bert-large-uncased, silver, Longman Defining Vocabulary)
A sparse contextual word encoder whose output vocabulary is restricted to the Longman Defining Vocabulary. Given a sentence with one word marked, it returns a sparse vector describing that word in that context — but only 1,897 of the 30,524 vocabulary dimensions (6.2%) can ever be non-zero, and each is a basic-English defining word.
The effect is that a vector reads as a definition:
"He sat on the <t> bank </t> of the river" -> bank shore bed edge water wall river face
"She withdrew the money from her <t> bank </t>" -> bank money account pay check payment trust debt
"The soldiers began their <t> charge </t>" -> charge attack march offensive advance defense battle
"There is no <t> charge </t> for delivery" -> charge cost tax payment pay rate price ticket
The restriction is trained, not applied afterwards: the mask is in place during training, so the gradients, the regularizers and the similarity all live in the restricted subspace.
The target word must be wrapped in <t> … </t>. Pooling runs over the tokens
between those markers only.
Usage
from sentence_transformers import SparseEncoder
import torch
model = SparseEncoder("jadermcs/lexsplade-bert-large-silver-ldv", trust_remote_code=True)
sents = [
"He sat on the <t> bank </t> of the river and watched the water.",
"She withdrew the money from her <t> bank </t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()
print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1)) # ~0.17
top = v[0].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['bank', 'shore', 'bed', 'edge', 'water', 'wall', 'river', 'face']
trust_remote_code=True is required. The restriction lives in the custom
pooling module (modeling_lexsplade.py, ids in
1_TargetWordSpladePooling/config.json). Loading the weights as a plain
AutoModelForMaskedLM and pooling yourself will produce unrestricted vectors,
which is not what this model was trained to emit.
Keep the marked word inside max_seq_length; if the markers are truncated away
the pooling falls back to the whole sentence.
The vocabulary
Built from the Longman Dictionary of Contemporary English defining vocabulary — the ~2,000 basic words in which every LDOCE definition is written. Three filters:
| step | dropped | remaining |
|---|---|---|
| published list, single-word entries | — | 2,188 |
must be one whole-word token in the BERT vocabulary (advertise → advert ##ise has no dimension of its own) |
63 | 2,125 |
| no stopwords or punctuation | 228 | 1,897 |
Shipped in this repo as longman_defining_vocabulary.txt (the word list) and
ldv_vocabulary.json (words plus the token ids the mask uses). The word list is
reproduced from the published LDOCE defining vocabulary for research use;
the dictionary and its defining vocabulary are the work of Longman/Pearson.
Training
| base model | google-bert/bert-large-uncased |
| training data | 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train) |
| objective | AnglE loss on the sparse target-word vectors, with a SPLADE FLOPS regularizer and a Barlow Twins term on the dense span representation |
| output space | 1,897 LDV dimensions, masked throughout training |
| seed | 42 |
Evaluation
MCL-WiC, English. AP is average precision; accuracy is at the best threshold.
| split | accuracy | AP |
|---|---|---|
| test | 90.70 | 96.16 |
| dev | 90.90 | 95.36 |
Mean nonzero dimensions per usage: 118 of 30,524 — roughly 60% fewer than the unrestricted model, at the same MCL-WiC AP (96.16 vs 96.30).
Variants
jadermcs/lexsplade-bert-large-silver-ldv(this model) — 1,897 LDV dimensions.jadermcs/lexsplade-bert-large-silver— identical training, full 30,524-dim output.