ModernBERT-Large-Instruct-WSD
Word sense disambiguation as multiple choice over WordNet (omw-en:1.4) definitions with a masked-LM answer
slot. Used by https://github.com/sign/word-sense-disambiguation (the WSDModernBertForMaskedLM class there
adds a 128-way answer-letter decoder; answer_letters.json lists the letters).
This revision (2026-09-04, "c3")
Fine-tuned from answerdotai/ModernBERT-Large-Instruct on: generated sentences for 97k synsets, WordNet's own
example sentences (minus a 5,000-example held-out slice), SemCor (222k instances, detokenized) and the
Princeton WordNet Gloss Corpus (manual tags). Cosine schedule, label smoothing 0.1, weight decay 0.01,
lr 2e-5, 2 epochs, batch 64, fp32 weights with bf16 autocast. Weights are stored in bf16.
Trained with the compact prompt template (the sentence with the target word marked, one definition per option
letter, a "none of the above" letter, then [unused0] [MASK]; 16% fewer tokens and ~20% faster than the earlier
wording). The repository code from PR #49 on builds exactly this prompt; earlier code/models are not compatible.
| Benchmark | previous revision (w4, earlier template) | this revision (c3) |
|---|---|---|
| WordNet held-out slice (5,000 examples, seed 42) | 78.6% | 78.3% |
| SemEval "ALL" (Senseval-2/3, SemEval-07/13/15; 7,247 instances, any gold key) | 80.7% | 80.6% |
| Throughput, one H100, same pipeline | 3,614 prompts/s | 4,326 prompts/s |
The previous revision over-predicted "none of the above" on long natural sentences; this one almost never does.
Full recipe and sweep configs: training/ in the repository.