ConfliBERT-cont-cased-squad2
snowood1/ConfliBERT-cont-cased fine-tuned for extractive question answering on SQuAD 2.0 with unanswerable
questions included, so the model can abstain (predict "no answer") when the context does not contain the answer.
This replaces the earlier salsarra/*-SQuAD* checkpoints, which were trained without unanswerable examples and never abstain
(null rate < 1% on SQuAD 2.0 dev).
Results (dev sets, official SQuAD scripts, null threshold 0)
| EM | F1 | HasAns F1 | NoAns F1 | best F1 (threshold) | |
|---|---|---|---|---|---|
| SQuAD 2.0 | 63.3 | 67.0 | 85.6 | 48.4 | 75.6 (-5.8) |
| SQuAD 1.1 | 79.1 | 86.3 |
Note: at threshold 0 this model under-abstains (NoAns F1 48); set null_score_diff_threshold to about -5.8 for F1 75.6.
Training
2 epochs on the full SQuAD 2.0 train split (130,319 questions), AdamW lr 3e-5, batch 32, linear schedule with 10% warmup,
fp16, max_length 384, doc_stride 128, standard HF run_qa label preparation (unanswerable -> [CLS]). Single RTX 4090, ~14 min.
Usage
from transformers import pipeline
qa = pipeline("question-answering", model="shreyasmeher/ConfliBERT-cont-cased-squad2", handle_impossible_answer=True)
qa(question="Who claimed responsibility?", context="A convoy was attacked near Gao on Tuesday. No group has claimed responsibility.")
# -> empty answer with a high null score
Or from R with conflibertR.
Citation
Hu, Y., Hosseini, M., Skorupa Parolin, E., Osorio, J., Khan, L., Brandt, P., & D'Orazio, V. (2022). ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence. NAACL 2022. https://doi.org/10.18653/v1/2022.naacl-main.400