AbsoluteZero NLI 2B

Observable NLI with think traces

Provide a text (premise) and a statement (hypothesis), and it returns calibrated probabilities that the text entails, contradicts, or is neutral to the statement. One model answers in two ways:

• Fast mode: a single forward pass and a classifier head (similar to cross-encoders).
• Think mode: the model writes a short reasoning trace, then the scoring head reads it and gives the verdict. Use it on the hard cases, such as multi-step logic, quantifiers and negation, long contracts and policies. Traces are short (~55 tokens on average on ANLI, ~220 on first-order-logic problems).

Benefits of observation

Typical zero-shot encoder models cannot provide information that elucidates why they chose their classification. Think traces enable users to inspect and determine how to augment hypotheses to fix or improve classification. This could be done in conjunction with automation for grouping classification failures and using an LLM to test changes to the hypothesis against a holdout set.

Examples

1.9B parameters · documents up to 32k tokens · #2 on the OpenNLI Skill Index (11-09-26, fast mode) · Apache-2.0

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-2b", trust_remote_code=True)
text = "My card was charged twice for the same order."
labels = ["billing", "shipping", "account access"]
probs = model.predict([(text, f"This text is about {l}.") for l in labels], apply_softmax=True)
print({l: round(float(p[0]), 3) for l, p in zip(labels, probs)})  # p[0] = P(entailment)
# {'billing': 0.976, 'shipping': 0.002, 'account access': 0.002}

More in Think Mode, Recommended Applications and the 45-dataset evaluation.

---

Table of Contents

• NLI Evaluation Results
• Usage
• Think Mode
• Recommended Applications
• Extended Evaluation (45 datasets)
• Training Details
• Limitations
• Acknowledgments
• Citation

---

NLI Evaluation Results

Comparison on the OpenNLI leaderboard (v0.2.8), fast mode.

^* trained on one or more OpenNLI test sets; those scores are replaced by estimates, as on the leaderboard.

---

Usage

Label Map:

• entailment: 0
• neutral: 1
• contradiction: 2

Fast mode (Sentence Transformers)

pip install -U sentence-transformers

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/AbsoluteZero-nli-2b", trust_remote_code=True)
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile floor.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
]

logits = model.predict(pairs)  # shape: (3, 3)
for (prem, hyp), row in zip(pairs, logits):
    print(f"[{id2label[int(np.argmax(row))]}]  Premise: {prem}  |  Hypothesis: {hyp}")

The model reads each pair in its training layout (Premise: … / Hypothesis: … / <think></think> / Relation:); the tokenizer shipped with the model builds it, so pass plain (premise, hypothesis) pairs. Inputs are right-padded; the head reads each row's last token.

Scoring multiple hypotheses on one text. The model is causal and the premise comes first, so for tagging or checking many claims against one document, score_many encodes the premise once and scores every hypothesis from that cached state: the same predictions as scoring pair by pair, ~10x faster for 50 labels on a ~500-token document.

doc = "My card was charged twice for the same order, and support hasn't replied in a week."
labels = ["billing", "shipping", "account access", "customer support"]
logits = model.model.score_many(model.tokenizer, doc, [f"This text is about {l}." for l in labels])
print({l: round(p, 3) for l, p in zip(labels, logits.softmax(-1)[:, 0].tolist())})  # P(entailment) per label
# {'billing': 0.965, 'shipping': 0.002, 'account access': 0.002, 'customer support': 0.991}

---

Think Mode

Think mode is a method on the loaded model; it returns the verdict, its probabilities and the trace.

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-2b", trust_remote_code=True)
pairs = [
    ("The glass fell off the counter and shattered on the tile floor.", "The glass broke when it hit the floor."),
    ("The store opens at 9 a.m. every day.", "The store opens at 7 a.m. on weekdays."),
    ("A researcher presented results at the conference.", "The presentation won the best paper award."),
]
for r in model.model.think(model.tokenizer, pairs):
    print(r["label"], "|", r["trace"])

entailment | Shattering on the floor means it broke upon hitting the floor, so the premise entails the hypothesis.
contradiction | The premise states the store opens at 9 a.m. daily, while the hypothesis states it opens at 7 a.m. on weekdays. These times conflict, so the premise contradicts the hypothesis.
neutral | Presenting results at a conference does not indicate whether the presentation won an award, so the premise neither guarantees nor rules out the hypothesis.

Think vs. fast (test sets: n=300 pairs each; ANLI dev: full dev splits):

Both modes have comparable accuracy with some variation. Confidence is calibrated for fast mode, while adding think traces will make the classification head certain toward the think trace, so confidence cannot be used in that mode.

---

Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Less suited: financial and general legal verification (findver, legal_domain), and adversarial reasoning (anli_r3, nope_adv).

---

Extended Evaluation (45 datasets)

Accuracy (fast mode) on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. DocNLI and VitaminC are evaluated on fixed random subsets of 5,000 and 10,000 pairs; FRANK is scored per summary. — = not on the same items.

Macro average over the 45 datasets: 0.8264 (4B: 0.8383).

---

Training Details

This model is a fine-tune of Qwen/Qwen3.5-2B (text backbone), trained in two stages.

Stage 1: distillation. Full fine-tune (all weights) & two classifier heads, trained on the FineCat-NLI-Reason compilation relabelled by an NLI fine-tune of Qwen3.8-27B.

Stage 2: reinforcement learning on think mode (GRPO). Think mode was further trained with 32 sampled traces per prompt, reward = correctness of the verdict with a soft overlong penalty, a repetition penalty and a floor that discourages trace collapse.

Stage 2 made think mode much shorter at equal or better accuracy. On first-order-logic validation problems, accuracy 0.792 → 0.806 with traces cut from ~365 to ~217 tokens on average.

The released weights are the stage-2 GRPO LoRA merged into the stage-1 weights, stored in bfloat16.

---

Limitations

• English only.
• Throughput: ~350 pairs/s on short pairs (~90 on long documents) in fast mode; use the FineCat encoders for very high volume or tight latency. Think mode generates a trace per pair (~1-10 pairs/s with transformers, depending on trace length; vLLM is faster).
• Trained on inputs up to 8k tokens; scored at up to 32k, but accuracy on documents longer than 8k is not separately tested.
• Weaker on financial and general legal verification (findver, legal_domain) and adversarial reasoning (anli_r3, nope_adv).

---

Acknowledgments

We thank the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{absolutezero-2026,
  title = {AbsoluteZero: Observable NLI},
  author = {Lee Miller},
  year = {2026},
  howpublished = {\url{https://huggingface.co/dleemiller/AbsoluteZero-nli-2b}}
}