BERT-base-uncased Fine-Tuned for Extractive Question Answering (SQuAD v1.1)

This repository contains the weights and tokenizer for bert-base-uncased fully fine-tuned on SQuAD v1.1 (Stanford Question Answering Dataset) for extractive question answering.

The model was adapted using end-to-end full fine-tuning with a differential two-learning-rate optimizer scheme as part of an academic comparative study on BERT adaptation paradigms.

---

Model Details

• Base Architecture: BERT-base (bert-base-uncased)
• Model Type: BertForQuestionAnswering
• Layers: 12 Transformer encoder blocks
• Hidden Dimension: 768
• Attention Heads: 12
• Parameters: 108,893,186 trainable parameters (~110M total)
• Framework: PyTorch & Hugging Face transformers

---

Training Data & Preprocessing

The model was trained on a deterministic subsample of SQuAD v1.1 (rajpurkar/squad):

• Subsample Size: 15,000 question-context pairs sampled deterministically with seed 42 from the training split.
• Sliding Window Strategy: Long contexts exceeding BERT's sequence limit were tokenized using a sliding window:
  • Maximum sequence length: 384 tokens
  • Document stride: 128 tokens
  • Truncation: only_second (context tokens truncated, questions kept intact)
  • Unanswerable / out-of-slice spans: Routed to index [CLS] (position 0)
• Windowed Features: The 15,000 source examples expanded to 15,141 training feature windows.

---

Evaluation Results

Evaluation was performed on the complete official SQuAD v1.1 validation split (10,570 examples, spanning 10,753 feature windows) using an optimized post-processing pipeline (top-15 start/end logit pruning, maximum answer span length of 30 tokens):

---

Intended Use

• Task: Extractive question answering in English. Given a question and a reference passage, the model extracts the continuous sub-string (span) that answers the question.
• Input: A query string (question) and a text passage (context).

Quickstart / How to Use

Depending on your version of transformers, choose one of the two options below:

Option A: Using pipeline (with transformers < 5)

Note: If you are using this method, ensure your environment has pip install "transformers<5".

from transformers import pipeline

# Initialize the pipeline directly from repository
qa_model = pipeline(
    task="question-answering",
    model="elisabetconese/Extractive-QA_BERT_body"
)

question = "Where was Super Bowl 50 played?"
context = "Super Bowl 50 took place at Levi's Stadium in Santa Clara, California."

# Run inference
prediction = qa_model(question=question, context=context)

print(f"Answer: {prediction['answer']}")

Option B: Direct Model Inference (transformers >= 5 or manual control)

Note: In transformers >= 5.0.0, the legacy question-answering pipeline was deprecated/removed. Use this approach to run inference directly with PyTorch tensors.

import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

model_id = "elisabetconese/Extractive-QA_BERT_body"

# Load tokenizer and model directly
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForQuestionAnswering.from_pretrained(model_id)

context = "Python was created by Guido van Rossum and first released in 1991."
question = "Who created Python?"

# Tokenize inputs, returning PyTorch tensors
inputs = tokenizer(question, context, return_tensors="pt")

# Perform inference without calculating gradients
with torch.no_grad():
    outputs = model(**inputs)

# Extract highest probability tokens for start and end positions
start_index = torch.argmax(outputs.start_logits)
end_index = torch.argmax(outputs.end_logits) + 1

# Decode the predicted tokens back to a string
predict_answer_tokens = inputs.input_ids[0, start_index : end_index]
answer = tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)

print(f"Extracted Answer: {answer}")

---

Limitations

1. Extractive Only: The model identifies spans verbatim within the context passage. It cannot synthesize, paraphrase, or generate abstractive responses.
2. SQuAD v1.1 Constraint (Answer-Present): SQuAD v1.1 assumes all questions have an answer in the provided text. The model does not support unanswerable question detection.
3. Context Length Truncation: While sliding window inference allows covering long documents, answer spans that cross window seams or exceed 384 tokens without stride coverage may suffer degradation.
4. Language & Casing: The model was pretrained on lowercased English text (uncased) and is unsuitable for multilingual texts or applications sensitive to letter casing.

---

References

1. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 §5.3.
2. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250.
3. Hugging Face Documentation. Fine-tuning a pretrained model. Hugging Face Transformers Training Documentation.