DeBERTa-v3-base span-based coreference resolution (WinoGrande)
End-to-end span-based coreference model (mention detection + antecedent ranking) with a
microsoft/deberta-v3-base encoder, fine-tuned on coref-data/winogrande_coref.
The coreference layers are implemented from scratch in PyTorch. This is not a sequence
classification head.
Architecture
[CLS] document [SEP] -> DeBERTa-v3-base (184M) -> h_t (768-d)
candidate span i (word-level enumeration, mapped to subwords via offset mapping)
g_i = [h_start ; h_end ; attn_pool(h_start..h_end) ; phi_width] (2324-d)
s_m(i) = FFNN_m(g_i) mention score
prune to k = min(30, ceil(0.4 * #words)) spans, restored to document order
s_a(i,j) = FFNN_a([g_i ; g_j ; g_i * g_j ; phi_dist(i,j)]) j < i only
s(i,j) = s_m(i) + s_m(j) + s_a(i,j), s(i,eps) = 0 dummy antecedent
y_i = argmax_{y in {eps} U {j<i}} s(i,y) -> links -> union-find clusters
| Component | Setting |
|---|---|
| Candidate spans | width ≤ 10 words, within one sentence, no punctuation-only boundary word |
| Width / distance embeddings | 20-d each; distance buckets 0,1,2,3,4,5–7,8–15,16–31,32–63,64+ |
| FFNNs | 1 hidden layer, 1024 units, GELU, dropout 0.3 |
| Task parameters | 9.5M |
| Singleton decoding | a span with no antecedent is a mention iff σ(s_m) > 0.5 |
Training objective
With Y(i) = {eps} ∪ {j < i} and P(y | i) = softmax_y s(i, y):
L = − 1/B Σ_docs [ Σ_i log Σ_{y ∈ GOLD(i)} P(y | i)
+ λ Σ_s ( z_s log σ(s_m(s)) + (1 − z_s) log(1 − σ(s_m(s))) ) ]
GOLD(i) is the set of kept spans from i's gold cluster that precede i, or {eps} if
there are none (non-mentions, first mentions, singletons). z_s indicates that candidate
s is a gold mention, and λ = 1. The marginal likelihood spreads probability mass over
all correct antecedents. The auxiliary mention loss is needed because the dataset annotates
singletons, which the antecedent loss alone cannot tell apart from non-mentions.
Data
| Split | Source | Documents |
|---|---|---|
| train | HF train minus dev groups |
34,229 |
| dev (model selection) | 5% of HF train twin groups (seed 42), grouped by WinoGrande twin id |
1,805 |
| test | HF validation |
1,117 |
Dataset revision: e5a194ba9c53da01acfb8c79589811fb87d7b9cf. Every document has one
2-mention chain (antecedent + pronoun) and one singleton (the distractor). Gold spans are
[sentence, start, end], with sentence-local word indices and inclusive ends. 99.91% of
training gold mentions are reachable as candidate spans. The other 93 have a punctuation
boundary word, mostly annotation noise such as ", sneakers". No document is truncated
(max 45 subwords).
Training hyper-parameters
| Parameter | Value |
|---|---|
| Optimizer | AdamW (fused), weight decay 0.01 |
| LR encoder / task layers | 1e-5 / 2e-4 |
| Schedule | linear, 10% warmup |
| Batch size | 16 documents, no accumulation |
| Epochs | up to 10, early stopping (patience 3 on dev CoNLL F1) |
| Gradient clipping | 1.0 |
| Precision | bf16 autocast, fp32 master weights |
| Seed | 42 |
| Hardware | 1× RTX 5060 Ti 16GB |
Evaluation
Corpus-level MUC, B³, CEAF-e, CEAF-m, CoNLL F1 = mean(MUC, B³, CEAF-e). Singletons are kept (the dataset annotates them). CoNLL F1 without singletons is also reported. Mention detection uses exact boundaries. Antecedent accuracy counts gold anaphors whose predicted antecedent belongs to their gold cluster.
Evaluation split: official validation split (1,117 docs)
| Metric | P | R | F1 |
|---|---|---|---|
| Mention detection | 97.44 | 97.85 | 97.65 |
| MUC | 76.08 | 76.01 | 76.04 |
| B³ | 90.06 | 90.38 | 90.22 |
| CEAF-e | 89.79 | 90.40 | 90.09 |
| CEAF-m | 90.22 | 90.60 | 90.41 |
| CoNLL F1 (with singletons) | 85.45 | ||
| CoNLL F1 (singletons removed) | 81.93 |
| Antecedent metric | Value |
|---|---|
| Gold anaphors | 1117 |
| Anaphor detection rate | 99.37 |
| Antecedent accuracy | 76.01 |
| Resolution accuracy | 76.01 |
Usage
The model needs the code files in this repo (model.py, config.py, preprocessing.py, dataset.py).
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("thealper2/deberta-v3-base-winogrande-coref")
sys.path.insert(0, path)
from model import CoreferenceModel
model = CoreferenceModel.from_pretrained(path, device="cuda")
result = model.predict(text="Sarah was a much better surgeon than Maria so she always got the harder cases.")
# result = {"mentions": [...], "clusters": [...], "antecedents": [...]}, character offsets, end exclusive
Limitations
- Trained only on short, single-paragraph WinoGrande documents (≤ 3 sentences, 2 candidate entities).
It is not evaluated on long documents; inputs are limited to
max_seq_length = 256subwords. - Gold mentions are only the two candidates and the pronoun. Other entities in the text are labelled as non-mentions, so mention detection is dataset-specific.
- The raw-text inference word splitter is regex-based (PTB-like clitic splitting); training used stanza tokens.