Ammu-1.0-OffOnTopic-Guardrail-small

Off On Topic Guardrail Classifier | ModernBERT Architecture | 68M Parameters | 90.35% Macro-F1

Overview

Ammu-1.0-OffOnTopic-Guardrail-small is a compact, high-efficiency cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to the assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

The model achieves an overall Macro-F1 score of 90.35% on a curated holdout benchmark of 4,008 test instances, substantially outperforming frontier LLMs such as DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +8.40% and +11.23% Macro-F1 points respectively, while maintaining a tiny 68M parameter footprint suitable for edge and latency-critical production deployments.

Key Features

• High Performance: Achieves a 90.35% overall Macro-F1 score (96.37% Easy, 88.93% Medium, 89.44% Hard), outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) across every difficulty tier.
• ModernBERT Backbone: Built upon the ModernBERT architecture with 68M parameters, offering ultra-fast inference and minimal memory overhead.
• Extensive Domain Taxonomy: Trained on balanced, high-signal data spanning 336 specialized domains, 49 difficulty nuance profiles, 19 prompt interaction formats, 29 user personas, 8 linguistic registers, and 9 tones.
• Sub-5ms Latency & Zero Token Cost: Provides deterministic upfront filtering with minimal latency overhead, bypassing expensive LLM inference passes for out-of-scope user interactions.

Use Case

• Upfront Domain Boundary Guardrail: Intercepts off-topic queries, prohibited tasks, and adversarial domain escapes before queries reach expensive LLM generation passes.
• Semantic Routing for Agentic Workflows: Routes in-scope domain queries to specialized tools, retrieval systems, or agent nodes while gracefully declining irrelevant requests.
• Cost & Latency Optimization: Prevents wasted computation, token budget exhaustion, and unnecessary database lookups caused by irrelevant or out-of-domain user interactions.

Usage

# First install the Hugging Face transformers library
!pip install transformers torch

from transformers import pipeline

# Initialize the Off On Topic guardrail classification pipeline
classifier = pipeline(
    "text-classification",
    model="kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-small",
    device=0  # Set to 0 for GPU, or -1 for CPU
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I implement a custom async context manager using contextlib in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Can you configure a Redux store with reducers and middleware in React JavaScript?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference
results = classifier(queries)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

"""
### Output
User Prompt: How do I implement a custom async context manager using contextlib in Python 3.12?
Prediction : On-Topic (Confidence: 0.8804)

User Prompt: Can you configure a Redux store with reducers and middleware in React JavaScript?
Prediction : Off Topic (Confidence: 0.7228)

User Prompt: Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?
Prediction : On-Topic (Confidence: 0.9318)

User Prompt: What are the health benefits of a Mediterranean diet and how many calories are in olive oil?
Prediction : Off Topic (Confidence: 0.9993)

User Prompt: I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?
Prediction : Off Topic (Confidence: 1.0000)
"""

Model Evaluation

The model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All models were evaluated under identical conditions in FP32 precision.

Ammu Cross-Encoder Family Leaderboard

Comparison with Frontier LLMs

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs with reasoning disabled to benchmark direct classification capability:

Key Takeaway: Ammu-1.0-OffOnTopic-Guardrail-small (68M) outperforms DeepSeek V4 Flash 0731 by +8.40% and GPT-6-luna by +11.23% overall Macro-F1. The performance difference is especially pronounced on Medium (+13.05% over DeepSeek, +17.30% over GPT-6) and Hard (+8.02% over DeepSeek, +10.32% over GPT-6) boundary queries, while running with sub-5ms latency and zero API cost.

Limitations

• Cross-Encoder Requirement: The model is a paired cross-encoder that evaluates user_prompt strictly with respect to a provided system_prompt. It cannot classify user queries in isolation without a reference system prompt.
• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.
• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-offontopic-guardrail-small-2026,
  title = {Ammu-1.0-OffOnTopic-Guardrail-small: Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-small}
}