🥬 ModernBERT-base-32k Hallucination Detector (Combined)

A hallucination detection model fine-tuned on RAGTruth + HaluEval datasets using extended 32K context ModernBERT.

Model Description

This model detects hallucinations in LLM-generated text by classifying each token as either Supported (grounded in context) or Hallucinated (not supported by context).

Key Features

• 32K Context Window: Built on llm-semantic-router/modernbert-base-32k with YaRN RoPE scaling
• Multi-Dataset Training: Trained on RAGTruth (~13.5K) + HaluEval (~38K) = 48K+ samples
• Token-Level Classification: Identifies specific spans that are hallucinated
• RAG Optimized: Trained on diverse RAG benchmarks for broad applicability

Performance

Evaluated on RAGTruth test set (2,700 samples):

Key Results

• ✅ Beats LettuceDetect BASE by +1% on example-level F1
• ✅ 3.5x more training data (RAGTruth + HaluEval)
• ✅ Better token-level F1 than RAGTruth-only model (53.37% vs 51.47%)
• ✅ More diverse training - generalizes better to different task types

Related Model

• modernbert-base-32k-haldetect - Trained on RAGTruth only (77.49% F1)

Training Data

HaluEval was converted from document-level to span-level annotations using NLI (DeBERTa-v3-mnli-fever-anli) and normalized to RAGTruth prompt format.

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

model_name = "llm-semantic-router/modernbert-base-32k-haldetect-combined"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Format: context + question + answer
text = """Context: The Eiffel Tower is located in Paris, France.
Question: Where is the Eiffel Tower?
Answer: The Eiffel Tower is located in London, England."""

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=8192)
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# 0 = Supported, 1 = Hallucinated

With LettuceDetect Library

from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="llm-semantic-router/modernbert-base-32k-haldetect-combined"
)

context = "The Eiffel Tower is located in Paris, France."
question = "Where is the Eiffel Tower?"
answer = "The Eiffel Tower is located in London, England."

spans = detector.predict(context, question, answer)

Training Configuration

base_model: llm-semantic-router/modernbert-base-32k
datasets:
  - data/ragtruth/ragtruth_data.json
  - data/halueval_spans_normalized/halueval_data.json
max_length: 8192
batch_size: 8
learning_rate: 1e-5
epochs: 6
loss: CrossEntropyLoss
scheduler: None (constant LR)

Model Variants

Citation

@misc{modernbert-32k-haldetect-combined,
  title={ModernBERT-base-32k Hallucination Detector (Combined)},
  author={llm-semantic-router},
  year={2025},
  url={https://huggingface.co/llm-semantic-router/modernbert-base-32k-haldetect-combined}
}

Acknowledgments

• Built on LettuceDetect framework
• Uses ModernBERT architecture
• Trained on RAGTruth and HaluEval datasets