Ammu-1.0-Nano-17M-ONNX-INT8

Quantized Off On Topic Guardrail Classifier | ModernBERT Architecture | 17M Parameters (INT8 Dynamic PTQ) | 85.26% Macro-F1

Overview

Ammu-1.0-Nano-17M-ONNX-INT8 is the Pareto-optimal post-training quantized (PTQ) ONNX INT8 variant of Ammu-1.0-OffOnTopic-Guardrail-nano, an ultra-compact edge cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to an assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

Generated using Dynamic INT8 Post-Training Quantization (PTQ) with Sensitive Layer & Classification Head Preservation, this model compresses the 17M distilled ModernBERT FP32 model from 64.5 MB down to 55.0 MB (14.7% storage reduction / 1.17x compression) while slashing single-query P50 CPU latency from 14.43 ms down to 11.88 ms (1.21x CPU speedup) and lifting throughput to 88.7 queries/sec (+8.6%).

Across the curated holdout benchmark of 4,008 test instances, the quantized model demonstrates near-lossless compression, retaining 99.6% of its FP32 baseline capability with an overall Macro-F1 score of 85.26% and a Hard Macro-F1 score of 84.91%. Operating with a minimal 17M parameter footprint, it decisively outperforms frontier LLMs including DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +3.31% and +6.14% Macro-F1 points respectively, with zero API token cost and a tiny memory footprint suited for edge devices and high-concurrency microservices.

Key Features

• Near-Lossless Quantized Macro-F1 Performance: Achieves an 85.26% overall Macro-F1 score and 84.91% Hard Macro-F1 score, retaining 99.6% of the FP32 baseline performance (only -0.32% delta) and outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%).

• 14.7% Reduced Memory Footprint: Compresses the model from 64.5 MB down to 55.0 MB (a 1.17x compression ratio), enabling deployment within ultra-lean worker processes on low-memory edge devices and micro instances.

• 1.21x CPU Latency Speedup: Slashes single-query P50 latency from 14.43 ms (FP32 baseline) down to 11.88 ms while boosting throughput from 81.7 to 88.7 queries/sec (+8.6%).

• Sensitive Layer & Head Preservation: Leverages data-driven activation sensitivity profiling to identify vulnerable transformer layers, preserving them alongside the classification head in full FP32 while quantizing remaining linear projections to signed INT8 (QuantType.QInt8).

Usage

# Install required dependencies & remove conflicting vision/diffusion packages (strictly as configured for Colab)
!pip install uv
!pip uninstall -y torchvision diffusers > /dev/null 2>&1 || true
!uv pip install --system -U "numpy>=2.1.0" "transformers>=4.48.0" "datasets>=3.0.0" "huggingface_hub>=0.28.0" "accelerate" "tabulate" "psutil" "optimum[onnxruntime]>=1.23.0" "onnx>=1.16.0" "onnxruntime>=1.19.0" "scikit-learn"

# Note: If upgrading packages in an active Colab session, restart session (Runtime -> Restart session) if prompted.

import sys
# Prevent incompatible diffusers from breaking Optimum's ONNX runtime loader
try:
    import diffusers
except Exception:
    sys.modules["diffusers"] = None

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

# Load the quantized ONNX INT8 guardrail model and tokenizer
model_id = "kalyan-ks/Ammu-1.0-Nano-17M-ONNX-INT8"

model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    file_name="model_quantized.onnx",
    provider="CPUExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the Hugging Face text-classification pipeline
classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I implement a custom async context manager using contextlib in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference (dynamic padding enabled)
results = classifier(queries, max_length=512, truncation=True, padding=True)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

Model Evaluation

The quantized model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All benchmarks were executed sequentially with ONNX Runtime on CPU with unified sequence limits (max_length=512).

Post-Training Quantization (PTQ) Progression: Ammu 1.0 Nano (17M)

Key Architectural Takeaways:

1. Why Sensitive Layer Preservation is Essential: In ultra-compact student models, naive full INT8 quantization causes total representation collapse (Macro-F1 dropping to 34.63%). Data-driven sensitivity profiling pinpointed the sensitive hidden layers. Retaining those sensitive layers alongside the classification head in full FP32 completely averts collapse, retaining 99.6% of baseline capability while preserving INT8 acceleration across the remaining layers.

2. Dynamic vs. Static INT8 on CPU: Static calibration approaches suffered from activation clipping degradation and higher CPU latency due to QDQ operator dequantize/quantize tensor overhead on variable-length textual sequences. Dynamic INT8 avoids activation clipping entirely.

3. Per-Channel Instability in ModernBERT: Per-channel weight quantization also suffered from representation collapse (34.63% Macro-F1), confirming that selective layer precision preservation is the only viable path to stable compression for ultra-compact models.

Ammu 1.0 Guardrail Quantized Family Leaderboard

Comparison of Pareto-optimal quantized models across the four Ammu 1.0 architectural tiers:

Comparison with Frontier LLMs and FP32 Baseline

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs and reasoning disabled to benchmark direct domain boundary classification capability:

Key Takeaway: Even after INT8 quantization, Ammu-1.0-Nano-17M-ONNX-INT8 outperforms DeepSeek V4 Flash 0731 by +3.31% and GPT-6-luna by +6.14% overall Macro-F1 points. The margin is particularly striking on Medium (+8.12% over DeepSeek, +12.37% over GPT-6) and Hard boundary queries (+3.49% over DeepSeek, +5.79% over GPT-6), delivering ultra-fast enterprise guardrailing at 11.88 ms CPU latency, 88.7 q/s throughput, and zero token cost.

Quantization Technical Specifications

• Quantization Framework: Microsoft ONNX Runtime Quantization Tools (onnxruntime.quantization) via Hugging Face Optimum.
• Quantization Approach: Dynamic INT8 Post-Training Quantization with Sensitive Layer & Head FP32 Preservation.
• Quantized Operators: MatMul, Gemm (attention projections, feed-forward linear layers outside sensitive layers).
• Excluded Operators: Data-driven sensitive intermediate transformer layers and classification head nodes (classifier, pooler, head, out_proj) preserved in full FP32.
• Weight Precision: Signed 8-bit Integer (QuantType.QInt8), per-tensor scaling.
• Activation Precision: Dynamic INT8 (quantized on-the-fly per request on CPU registers).
• Calibration Requirement: None (Dynamic PTQ requires zero calibration data).
• Target Runtime Execution Provider: CPUExecutionProvider.

Limitations

• CPU Execution Target: This INT8 dynamic ONNX artifact is specifically tailored for CPU inference via ONNX Runtime. For GPU acceleration, native TensorRT FP16/INT8 execution engines are recommended.

• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.

• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-off-on-topic-guadrail-nano-17m-onnx-int8-2026,
  title = {Ammu-1.0-Nano-17M-ONNX-INT8: Quantized Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-nano-ONNX-INT8}
}