EttinX Cross-Encoder: Natural Language Inference (NLI)

This cross encoder performs sequence classification for contradiction/neutral/entailment labels. This has drop-in compatibility with comparable sentence transformers cross encoders.

To train this model, I added teacher logits to the all-nli dataset dleemiller/all-nli-distill from the dleemiller/ModernCE-large-nli model. This significantly improves performance above standard training.

This 32m architecture is based on ModernBERT and is an excellent candidate for lightweight CPU inference.

---

Features

• High performing: Achieves 83.80% and 88.20% (Micro F1) on MNLI mismatched and SNLI test.
• Efficient architecture: Based on the Ettin-32m encoder design (32M parameters), offering faster inference speeds.
• Extended context length: Processes sequences up to 8192 tokens, great for LLM output evals.

---

Performance

---

Usage

To use EttinX for NLI tasks, you can load the model with the Hugging Face sentence-transformers library:

from sentence_transformers import CrossEncoder

# Load EttinX model
model = CrossEncoder("dleemiller/EttinX-nli-xs")

scores = model.predict([
    ('A man is eating pizza', 'A man eats something'),
    ('A black race car starts up in front of a crowd of people.', 'A man is driving down a lonely road.')
])

# Convert scores to labels
label_mapping = ['contradiction', 'entailment', 'neutral']
labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]
# ['entailment', 'contradiction']

---

Training Details

Pretraining

We initialize the `` weights.

Details:

• Batch size: 512
• Learning rate: 1e-4
• Attention Dropout: attention dropout 0.1

Fine-Tuning

Fine-tuning was performed on the dleemiller/all-nli-distill dataset.

Validation Results

The model achieved the following test set micro f1 performance after fine-tuning:

• MNLI Unmatched: 0.8380
• SNLI: 0.8820

---

Model Card

• Architecture: Ettin-encoder-32m
• Fine-Tuning Data: dleemiller/all-nli-distill

---

Thank You

Thanks to the Johns Hopkins team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models.

---

Citation

If you use this model in your research, please cite:

@misc{moderncenli2025,
  author = {Miller, D. Lee},
  title = {EttinX NLI: An NLI cross encoder model},
  year = {2025},
  publisher = {Hugging Face Hub},
  url = {https://huggingface.co/dleemiller/EttinX-nli-xxs},
}

---

License

This model is licensed under the MIT License.