TinyBERT_General_4L_312D-nli

[!CAUTION] This model has poor Natural Language Inference (NLI) performance probably due to its small size. For research and experimental use only.

• Base Model: huawei-noah/TinyBERT_General_4L_312D
• Task: Natural Language Inference (NLI)
• Framework: Hugging Face Transformers, Sentence Transformers

TinyBERT_General_4L_312D-nli is a fine-tuned NLI model that classifies the relationship between pairs of sentences into three categories: entailment, neutral, and contradiction. It enhances the capabilities of huawei-noah/TinyBERT_General_4L_312D for improved performance on NLI tasks.

Intended Use

TinyBERT_General_4L_312D-nli is ideal for research applications requiring understanding of logical relationships between sentences, including:

• Semantic textual similarity
• Question answering
• Dialogue systems
• Content moderation

Performance

TinyBERT_General_4L_312D-nli was trained on the sentence-transformers/all-nli dataset, achieving better than random results in sentence pair classification.

Performance on the MNLI matched validation set:

• Accuracy: 0.5911
• Precision: 0.68
• Recall: 0.60
• F1-score: 0.58

Training details

Training Details

• Dataset:

  • Used sentence-transformers/all-nli.

• Sampling:

  • 100 000 training samples and 10 000 evaluation samples.

• Fine-tuning Process:

  • Custom Python script with adaptive precision training (bfloat16).
  • Early stopping based on evaluation loss.

• Hyperparameters:

  • Learning Rate: 2e-5
  • Batch Size: 32
  • Optimizer: AdamW (weight decay: 0.01)
  • Training Duration: Up to 10 epochs

Reproducibility

To ensure reproducibility:

• Fixed random seed: 42
• Environment:
  • Python: 3.10.12
  • PyTorch: 2.5.1
  • Transformers: 4.44.2

Usage Instructions

Using Sentence Transformers

from sentence_transformers import CrossEncoder

model_name = "agentlans/TinyBERT_General_4L_312D-nli"
model = CrossEncoder(model_name)
scores = model.predict(
    [
        ("A man is eating pizza", "A man eats something"),
        (
            "A black race car starts up in front of a crowd of people.",
            "A man is driving down a lonely road.",
        ),
    ]
)

label_mapping = ["entailment", "neutral", "contradiction"]
labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]
print(labels)

Using Transformers Library

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "agentlans/TinyBERT_General_4L_312D-nli"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

features = tokenizer(
    [
        "A man is eating pizza",
        "A black race car starts up in front of a crowd of people.",
    ],
    ["A man eats something", "A man is driving down a lonely road."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)

model.eval()
with torch.no_grad():
    scores = model(**features).logits
    label_mapping = ["entailment", "neutral", "contradiction"]
    labels = [label_mapping[score_max] for score_max in scores.argmax(dim=1)]
    print(labels)

Limitations and Ethical Considerations

TinyBERT_General_4L_312D-nli may reflect biases present in the training data. Users should evaluate its performance in specific contexts to ensure fairness and accuracy.

More importantly, this model has poor performance on the NLI task.

Conclusion

TinyBERT_General_4L_312D-nli is a model for NLI tasks, enhancing huawei-noah/TinyBERT_General_4L_312D's capabilities with straightforward integration into existing frameworks. It aids developers in building intelligent applications that require nuanced language understanding.