Ammu-1.0-OffOnTopic-Guardrail-nano

Off On Topic Guardrail Classifier | ModernBERT Architecture | 17M Parameters | 87.31% Macro-F1

Overview

Ammu-1.0-OffOnTopic-Guardrail-nano is an edge-optimized, ultra-compact cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to the assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

The model achieves an overall Macro-F1 score of 87.31% on a curated holdout benchmark of 4,008 test instances, outperforming frontier LLMs such as DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +5.36% and +8.19% Macro-F1 points respectively, while operating with a minimal 17M parameter footprint specifically designed for real-time edge, CPU-only, or high-throughput microservice architectures.

Quantized (INT8) Model

Key Features

• High Performance at Micro Scale: Achieves an 87.31% overall Macro-F1 score (91.77% Easy, 86.55% Medium, 86.39% Hard), outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) across every difficulty tier.
• ModernBERT Backbone: Built upon the ModernBERT architecture with 17M parameters, offering lightning-fast execution and a negligible memory footprint.
• Extensive Domain Taxonomy: Trained on balanced, high-signal data spanning 336 specialized domains, 49 difficulty nuance profiles, 19 prompt interaction formats, 29 user personas, 8 linguistic registers, and 9 tones.
• Sub-2ms Latency & Zero Token Cost: Provides deterministic upfront filtering with minimal latency overhead, bypassing expensive LLM inference passes for out-of-scope user interactions.

Use Case

• Upfront Domain Boundary Guardrail: Intercepts off-topic queries, prohibited tasks, and adversarial domain escapes before queries reach expensive LLM generation passes.
• Semantic Routing for Agentic Workflows: Routes in-scope domain queries to specialized tools, retrieval systems, or agent nodes while gracefully declining irrelevant requests.
• Cost & Latency Optimization: Prevents wasted computation, token budget exhaustion, and unnecessary database lookups caused by irrelevant or out-of-domain user interactions.

Usage

# First install the Hugging Face transformers library
!pip install transformers torch

from transformers import pipeline

# Initialize the Off On Topic guardrail classification pipeline
classifier = pipeline(
    "text-classification",
    model="kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-nano",
    device=0  # Set to 0 for GPU, or -1 for CPU
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I reverse a string without using in-built function in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Can you configure a Redux store with reducers and middleware in React JavaScript?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference
results = classifier(queries)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

"""
### Output
User Prompt: How do I reverse a string without using in-built function in Python 3.12?
Prediction : On-Topic (Confidence: 0.8911)

User Prompt: Can you configure a Redux store with reducers and middleware in React JavaScript?
Prediction : Off Topic (Confidence: 0.8590)

User Prompt: Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?
Prediction : On-Topic (Confidence: 0.8382)

User Prompt: What are the health benefits of a Mediterranean diet and how many calories are in olive oil?
Prediction : Off Topic (Confidence: 0.8632)

User Prompt: I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?
Prediction : Off Topic (Confidence: 0.9965)
"""

Model Evaluation

The model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All models were evaluated under identical conditions in FP32 precision.

Ammu Cross-Encoder Family Leaderboard

Comparison with Frontier LLMs

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs with reasoning disabled to benchmark direct classification capability:

Key Takeaway: Ammu-1.0-OffOnTopic-Guardrail-nano (17M) outperforms DeepSeek V4 Flash 0731 by +5.36% and GPT-6-luna by +8.19% overall Macro-F1. The performance difference is especially pronounced on Medium (+10.67% over DeepSeek, +14.92% over GPT-6) and Hard (+4.97% over DeepSeek, +7.27% over GPT-6) boundary queries, while running with sub-2ms latency and zero API cost.

Limitations

• Cross-Encoder Requirement: The model is a paired cross-encoder that evaluates user_prompt strictly with respect to a provided system_prompt. It cannot classify user queries in isolation without a reference system prompt.
• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.
• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-offontopic-guardrail-nano-2026,
  title = {Ammu-1.0-OffOnTopic-Guardrail-nano: Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-nano}
}