DistilBERT fine-tuned on SQuAD 2.0

distilbert-base-uncased fine-tuned for extractive question answering on SQuAD 2.0. Given a passage and a question, it marks the words in the passage that answer the question, or says that the passage does not contain an answer.

A course project for CSCI 390 (Natural Language Processing and Applications) by David Falaye and Kosisochukwu Ogbuanya. Code, training notebook, evaluation and error analysis: https://github.com/Davidfaleye360/AI-Question-Answering-System

How to use it

from transformers import pipeline

qa = pipeline("question-answering", model="Ingsyling/distilbert-squad2")
result = qa(
    question="In what country is Normandy located?",
    context="The Normans gave their name to Normandy, a region in France.",
    handle_impossible_answer=True,   # allow "no answer"; the answer is then an empty string
)
print(result["answer"])   # France

This needs transformers 4.x (version 5 removed the question-answering pipeline). The project's own code (src/predict.py in the repository) does the same without the pipeline and also returns the top candidates and the no-answer score.

Results

SQuAD 2.0 dev set, 11,873 questions, scored with the official squad_v2 metric at the default no-answer threshold.

	Exact Match	F1	Questions
All	64.14	67.38	11,873
Answerable	65.69	72.17	5,928
Unanswerable	62.59	62.59	5,945

For comparison, measured with the same code:

Model	Exact Match	F1	Parameters	CPU time per question
This model	64.14	67.38	66 M	31 ms
`deepset/roberta-base-squad2`	79.90	82.91	124 M	57 ms
`distilbert-base-uncased` before fine-tuning (first 1,000 questions)	1.2	4.6	66 M

CPU time is the median over 200 dev questions on an Apple M2 laptop. Tuning the no-answer threshold raises this model's F1 to 68.08 (cross-validated over dev articles).

Training

Setting	Value
Data	SQuAD 2.0 training set, 130,319 questions (131,754 windows)
Window	384 tokens, 128 tokens of overlap
Epochs	2 (16,470 steps)
Batch size	16
Learning rate	3e-5, linear warm-up over the first 10% of steps, then linear decay
Weight decay	0.01
Precision	mixed (fp16)
Seed	42
Hardware and time	one Tesla T4 (Google Colab), 44 minutes

Unanswerable questions, and windows that do not contain the whole answer, are labelled with the [CLS] position. The training loss fell from 5.81 to about 0.9.

Limitations

It copies, it does not reason. The answer is always one continuous piece of the passage. Questions whose answer has to be put together from several sentences, and most "why" questions, are beyond it.
It is too willing to answer. 52% of its errors on the dev set are answers to unanswerable questions, typically questions that reuse the passage's wording and change one detail or reverse the meaning.
"Why" and "how" questions and long answers are weak: F1 is 59 for "why" questions and 51 for answers of ten or more words, against 85 for "how many".
Long documents need care. Reading a whole article in overlapping windows lowers F1 by 16 to 23 points with this model's default settings, mainly because it stops abstaining.
English Wikipedia only. It was trained on Wikipedia paragraphs, uncased, and has not been tested on other kinds of text or other languages.
Do not use it where a wrong answer has consequences (medical, legal, financial) without a person checking.

Licences

The base model is released under Apache 2.0. SQuAD 2.0 is released under CC BY-SA 4.0.

Loading…