GoingDeep/bert-base-uncased-finetuned-for-qnli

bert-base-uncased-finetuned-qnli

This model is bert-base-uncased fine-tuned on the QNLI subset of GLUE. Given a question and a sentence, it predicts whether the sentence contains the answer to the question.

Model Details

Model Description

• Developed by: GoingDeep
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: BERT with a sequence-classification head (2 labels)
• Language(s) (NLP): English
• License: apache-2.0
• Finetuned from model [optional]: bert-base-uncased

Model Sources [optional]

• Repository: https://github.com/SauravSG/Deep-Learning/blob/main/fine-tuning/QNLI_model.ipynb
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Input is a pair of texts in this order: the question first, then the sentence. The model outputs one of two labels:

• entailment (label 0): the sentence answers the question
• not_entailment (label 1): it does not

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

Not intended for non-English text, or for decisions that need a reliable judgement of whether a text answers a question. It was trained on a small subsample of one dataset.

Bias, Risks, and Limitations

• English only.
• Trained on Wikipedia-style question/sentence pairs, so it may be unreliable on other kinds of text.
• Trained on a 20,000-pair subsample of the training split, not the full set.
• Scores below are on the validation split, which was also used to select the best checkpoint, so they are slightly optimistic.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "GoingDeep/bert-base-uncased-finetuned-qnli"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

pairs = [
    ("What is the capital of France?", "Paris is the capital and largest city of France."),
    ("What is the capital of France?", "The Amazon is the largest rainforest in the world."),
]

for question, sentence in pairs:
    inputs = tokenizer(question, sentence, return_tensors="pt")
    with torch.inference_mode():
        logits = model(**inputs).logits
    pred_id = logits.argmax(dim=-1).item()
    print(f"{question} | {sentence} -> {model.config.id2label[pred_id]}")

Training Details

Training Data

A random 20,000-pair subsample (shuffled with seed 42) of the 104,743 QNLI training pairs.

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime:
• 3 epochs, batch size 16, learning rate 5e-5 (3,750 steps in total)
• Evaluated and saved every epoch
• Best checkpoint (lowest validation loss) loaded at the end

Speeds, Sizes, Times [optional]

About 21 minutes (3,750 steps) on a Google Colab GPU.

Evaluation

Testing Data, Factors & Metrics

Testing Data

GLUE QNLI validation split (5,463 pairs). The test split has no public labels.

Factors

[More Information Needed]

Metrics

Accuracy

Results

Validation loss was lowest after epoch 1 and then rose while training loss kept falling, a sign of overfitting. The published checkpoint is the one with the lowest validation loss, not the highest accuracy. The accuracy differences between epochs are small (about 30 examples out of 5,463).

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

Google Colab GPU

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]