Ammu-1.0-FactCheck-base

Fact-Check Need Classifier | ModernBERT Architecture | 150M Parameters | 96.56% Macro-F1

Overview

Ammu-1.0-FactCheck-base is a lightweight encoder model trained to classify user prompts into (FACT_CHECK_NEEDED) or does not (NO_FACT_CHECK_NEEDED). It is trained on a diverse, high-quality synthetic dataset of user prompts labeled as FACT_CHECK_NEEDED NO_FACT_CHECK_NEEDED. It achieves an overall Macro-F1 score of 96.56%, outperforming much larger 307M encoder baselines across all difficulty tiers (easy, medium and hard). This model can be used an upfront guardrail that directs queries to retrieval-augmented generation (RAG) pipelines or web search engines only when factual grounding is genuinely needed.

Key Features

• Achieves a 96.56% overall Macro-F1 score (98.91% Easy, 96.74% Medium, 95.23% Hard), outperforming the 307M parameter baseline by more than +27% Macro-F1 points.
• Built upon ModernBERT architecture with 150M parameters
• Trained on balanced synthetic data spanning 120 granular topics, 19 prompt interaction formats, and 29 user personas.

Use Case

• Serves as an efficient user prompt classifier that saves LLM token costs and latency by bypassing unnecessary search or retrieval for non-factual queries (creative writing, coding, ideation, math puzzles, and text transformations).

Usage

# First install the Hugging Face transformers library
!pip install transformers torch

from transformers import pipeline

# Initialize the Fact-Check classification pipeline
classifier = pipeline(
    "text-classification",
    model="kalyan-ks/Ammu-1.0-FactCheck-Need-base",
    device=0  # Set to 0 for GPU, or -1 for CPU
)

# Example input prompts
prompts = [
    "India is the most populous country in the world. Do you agree?",
    "Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?",
    "I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. My age?",
    "Kalyan KS is from India. Extract named entities from the given sentence.",
    "Extract named entities: Kalyan KS is from India.",
]

# Run classification inference
results = classifier(prompts)

# Display predictions
for prompt, res in zip(prompts, results):
    print(f"Prompt: {prompt}")
    print(f"Prediction: {res['label']} (Confidence: {res['score']:.4f})\n")

"""
###Output
Prompt: India is the most populous country in the world. Do you agree?
Prediction: FACT_CHECK_NEEDED (Confidence: 0.6767)

Prompt: Assume that I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. How old I am today?
Prediction: FACT_CHECK_NEEDED (Confidence: 0.9998)

Prompt: I was born 16 years 4 months 3 days exactly after Sachin Tendulkar was born. My age?
Prediction: FACT_CHECK_NEEDED (Confidence: 1.0000)

Prompt: Kalyan KS is from India. Extract named entities from the given sentence.
Prediction: NO_FACT_CHECK_NEEDED (Confidence: 1.0000)

Prompt: Extract named entities: Kalyan KS is from India.
Prediction: NO_FACT_CHECK_NEEDED (Confidence: 1.0000)
"""

Model Evaluation

This model was evaluated on a curated holdout benchmark test set of 4,214 balanced instances across three difficulty tiers (Easy, Medium, and Hard). Macro-F1 scores are reported below:

Limitations

• Language Scope: The model has been trained and evaluated predominantly on English language texts; classification accuracy on multilingual or code-switched inputs may be lower.
• Classification vs. Fact Verification: This model only predicts the need for factual verification based on the user's input intent; it does not check the truth value of factual claims or generate factual answers.
• Highly Ambiguous or Mixed Prompts: Prompts that seamlessly mix complex creative writing with subtle real-world factual claims in a single query may present borderline confidence scores and benefit from threshold tuning.

Citation

@misc{ammu-1.0-factcheck-need-base-2026,
  title = {Ammu-1.0-FactCheck-base: Fact-Check Need Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-FactCheck-Need-base}
}