FineCat-NLI dbv3 Small (DeBERTa-v3-xsmall)

Try it: live demo Space

Natural Language Inference (NLI): the Swiss Army knife of encoders

Provide a text (premise) and statement (hypothesis), and it returns calibrated probabilities that the text entails, contradicts, or is neutral to the statement. A multiple-choice question is one statement per option.

Examples

71M parameters (microsoft/deberta-v3-xsmall, 22M backbone) · inputs up to 512 tokens · ~7,500 pairs/s · Apache-2.0

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/finecat-nli-dbv3-s")
text = "My credit card was billed twice for one purchase."
labels = ["billing", "shipping", "account access"]
probs = model.predict([(text, f"This text is about {l}.") for l in labels], apply_softmax=True)
print({l: round(float(p[0]), 3) for l, p in zip(labels, probs)})  # p[0] = P(entailment)
# {'billing': 0.946, 'shipping': 0.003, 'account access': 0.07}

More in Recommended Applications and the 45-dataset evaluation.

---

Table of Contents

• NLI Evaluation Results
• Usage
• Recommended Applications
• Extended Evaluation (45 datasets)
• Training Details
• Acknowledgments
• Citation

---

NLI Evaluation Results

Comparison on the OpenNLI leaderboard (v0.2.5).

^* trained on one or more OpenNLI test sets; those scores are replaced by estimates, as on the leaderboard.

---

Usage

Label Map:

• entailment: 0
• neutral: 1
• contradiction: 2

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load the model and run inference.

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/finecat-nli-dbv3-s")
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
    ("It started raining heavily, so the match was postponed.",
     "The game was delayed due to weather."),             # E
    ("Every seat on the flight was taken.",
     "There were several empty seats on the plane."),     # C
]

logits = model.predict(pairs)  # shape: (5, 3)

# inputs over 512 tokens: truncate the text (premise) only, never the statement (hypothesis)
# logits = model.predict(pairs, processing_kwargs={"text": {"truncation": "only_first"}})

for (prem, hyp), row in zip(pairs, logits):
    pred_idx = int(np.argmax(row))
    pred = id2label[pred_idx]
    print(f"[{pred}]  Premise: {prem}  |  Hypothesis: {hyp}")

---

Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Less suited: financial and general legal verification (findver, legal_domain), clinical-trial eligibility (nli4ct), multi-document legal reasoning (control), and adversarial reasoning (anli_r3, nope_adv).

---

Extended Evaluation (45 datasets)

Accuracy on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. Inputs are capped at 512 tokens (premise truncated). DocNLI and VitaminC are evaluated on fixed random subsets of 1,000 and 10,000 pairs. FRANK (news-summary factuality) is scored per summary sentence: each sentence vs. its source article, labelled by the majority annotator judgment.

Macro average over the 45 datasets: 0.7685.

Training Details

This model is a fine-tune of microsoft/deberta-v3-xsmall, trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional NLI, fact verification and long document sources (2.05M pairs in total). The training uses logit distillation from an NLI finetune of Qwen3.8-27B.

Distillation loss looks like this: $$ \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(z^{(s)}, y) + \beta \cdot T^2 \cdot \mathrm{KL}\left(\sigma(z^{(t)}/T) ,|, \sigma(z^{(s)}/T)\right) \end{equation} $$

where \(z^{(s)}\) and \(z^{(t)}\) are the student and teacher logits, \(y\) are the ground truth labels, \(\alpha = 0.2\), \(\beta = 0.8\) and \(T = 1\).

Inputs are capped at DeBERTa-v3's 512-token pretraining length, truncating the premise only. The model was trained for 3 epochs at a constant learning rate followed by a 1-epoch cooldown, with fp32 master weights (the base checkpoint ships in fp16). The released weights are the soup greedy_val, selected on the validation/dev objective (never a test set), with the output temperature calibrated on the validation split. The learning rate was 3.8e-5: 0.85x the official DeBERTa-v3 MNLI rate, whose optimality a WSD learning-rate probe confirmed.

---

Acknowledgments

We thank Microsoft (DeBERTa-v3) and the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{nli-compiled-2025,
  title = {FineCat NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Refined compilation of 6 major NLI datasets}
}