Legal-BERT fine-tuned on ContractNLI with standard CE
Document-level NLI for Non-Disclosure Agreements (NDAs)
Given an NDA contract and a hypothesis about a standard provision, classify as:
| Metric | Score |
|---|---|
| Micro-F1 | 86.7% |
| Macro-F1 | 77.0% |
| Parameters | 110M |
| Loss | Standard Cross-Entropy |
| Model | Micro-F1 | Macro-F1 | Params | Notes |
|---|---|---|---|---|
| Rule-based baseline | 20.9% | 16.0% | — | Keyword matching |
| BERT-base (paper) | ~83% | — | 110M | ContractNLI paper reference |
| contractnli-legalbert-nda-weighted | 87.3% | 79.3% | 110M | highest accuracy |
| contractnli-legalbert-nda-standard | 86.7% | 77.0% | 110M | ← this model |
| contractnli-bert-nda-standard | 86.9% | 76.7% | 110M | paper reproduction |
| contractnli-bert-nda-weighted | 86.3% | 77.9% | 110M | |
| contractnli-distilbert-nda | 86.0% | 76.2% | 66M | fastest inference, recommended for production |
| ID | Provision |
|---|---|
| nda-1 | Explicit identification |
| nda-2 | Non-inclusion of non-technical information |
| nda-3 | Inclusion of verbally conveyed information |
| nda-4 | Limited use |
| nda-5 | Sharing with employees |
| nda-7 | Sharing with third-parties |
| nda-8 | Notice on compelled disclosure |
| nda-10 | Confidentiality of Agreement |
| nda-11 | No reverse engineering |
| nda-12 | Permissible development of similar information |
| nda-13 | Permissible acquirement of similar information |
| nda-15 | No licensing |
| nda-16 | Return of confidential information |
| nda-17 | Permissible copy |
| nda-18 | No solicitation |
| nda-19 | Survival of obligations |
| nda-20 | Permissible post-agreement possession |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
model_name = "Agreemind/contractnli-legalbert-nda-standard"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
hypothesis = "All Confidential Information shall be expressly identified by the Disclosing Party."
premise = "Section 2.1: Any information disclosed must be marked as Confidential..."
inputs = tokenizer(hypothesis, premise, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)
pred = ["Entailment", "Contradiction", "NotMentioned"][probs.argmax()]
print(f"Prediction: {pred} (confidence: {probs.max():.3f})")
nlpaueb/legal-bert-base-uncased@inproceedings{koreeda-manning-2021-contractnli,
title = "ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts",
author = "Koreeda, Yuta and Manning, Christopher",
booktitle = "Findings of EMNLP 2021",
year = "2021",
}
MIT
nlpaueb/legal-bert-base-uncased