Ammu-1.0-Tiny-32M-ONNX-INT8

Quantized Off On Topic Guardrail Classifier | ModernBERT Architecture | 32M Parameters (INT8 Dynamic PTQ) | 87.03% Macro-F1

Overview

Ammu-1.0-Tiny-32M-ONNX-INT8 is the Pareto-optimal post-training quantized (PTQ) ONNX INT8 variant of Ammu-1.0-OffOnTopic-Guardrail-tiny, an ultra-lightweight cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to an assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

Generated using Dynamic INT8 Post-Training Quantization (PTQ) with Sensitive Layer & Classification Head Preservation, this model compresses the 32M distilled ModernBERT FP32 graph from 122.4 MB down to 90.3 MB (26.2% storage reduction / 1.36x compression) while cutting single-query P50 CPU latency from 35.03 ms down to 31.50 ms (1.11x CPU speedup) and boosting throughput to 35.9 queries/sec (+28.7%).

Across the curated holdout benchmark of 4,008 test instances, the quantized model retains 96.7% of its FP32 baseline capability, achieving an overall Macro-F1 score of 87.03% and a Hard Macro-F1 score of 86.80%. Despite operating with an ultra-compact 32M parameter footprint, it decisively outperforms frontier LLMs including DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +5.08% and +7.91% Macro-F1 points respectively, with zero API token cost and low CPU memory requirements.

Key Features

• High Quantized Macro-F1 Performance: Achieves an 87.03% overall Macro-F1 score and 86.80% Hard Macro-F1 score, retaining 96.7% of the FP32 baseline performance and outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%).

• 26.2% Reduced Memory Footprint: Compresses the model from 122.4 MB down to 90.3 MB (a 1.36x compression ratio), substantially lowering memory overheads for fast microservice deployments.

• 1.11x CPU Latency Speedup: Slashes single-query P50 latency from 35.03 ms (FP32 baseline) down to 31.50 ms while boosting throughput from 27.9 to 35.9 queries/sec (+28.7%) on standard CPU environments without requiring specialized accelerator hardware.

• Sensitive Layer & Head Preservation: Leverages data-driven activation sensitivity profiling to identify vulnerable transformer layers, preserving them alongside the classification head in full FP32 while quantizing remaining linear projections to signed INT8 (QuantType.QInt8).

Usage

# Install required dependencies & remove conflicting vision/diffusion packages (strictly as configured for Colab)
!pip install uv
!pip uninstall -y torchvision diffusers > /dev/null 2>&1 || true
!uv pip install --system -U "numpy>=2.1.0" "transformers>=4.48.0" "datasets>=3.0.0" "huggingface_hub>=0.28.0" "accelerate" "tabulate" "psutil" "optimum[onnxruntime]>=1.23.0" "onnx>=1.16.0" "onnxruntime>=1.19.0" "scikit-learn"

# Note: If upgrading packages in an active Colab session, restart session (Runtime -> Restart session) if prompted.

import sys
# Prevent incompatible diffusers from breaking Optimum's ONNX runtime loader
try:
    import diffusers
except Exception:
    sys.modules["diffusers"] = None

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

# Load the quantized ONNX INT8 guardrail model and tokenizer
model_id = "kalyan-ks/Ammu-1.0-Tiny-32M-ONNX-INT8"

model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    file_name="model_quantized.onnx",
    provider="CPUExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the Hugging Face text-classification pipeline
classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I implement a custom async context manager using contextlib in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference (dynamic padding enabled)
results = classifier(queries, max_length=512, truncation=True, padding=True)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

Model Evaluation

The quantized model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All benchmarks were executed sequentially with ONNX Runtime on CPU with unified sequence limits (max_length=512).

Post-Training Quantization (PTQ) Progression: Ammu 1.0 Tiny (32M)

The complete benchmark progression across different post-training quantization techniques reveals the performance and efficiency tradeoffs for the Tiny 32M architecture:

Key Architectural Takeaways:

1. Why Sensitive Layer Preservation is Essential: In compact student models, representation capacity is concentrated across fewer layers. Quantizing all layers introduces activation drift that degrades boundary discrimination. Retaining the most sensitive intermediate layers alongside the classification head in full FP32 protects critical decision margins.

2. Dynamic vs. Static INT8 on CPU: Static calibration approaches suffered from activation clipping degradation and higher CPU latency due to QDQ operator dequantize/quantize tensor overhead on variable-length textual sequences. Dynamic INT8 avoids activation clipping entirely.

3. Per-Channel Instability in ModernBERT: Per-channel weight quantization introduced channel scaling discrepancies that degraded Macro-F1 to 79.15%, whereas per-tensor dynamic quantization maintained robust representation fidelity.

Ammu 1.0 Guardrail Quantized Family Leaderboard

Comparison of Pareto-optimal quantized models across the four Ammu 1.0 architectural tiers:

Comparison with Frontier LLMs and FP32 Baseline

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs and reasoning disabled to benchmark direct domain boundary classification capability:

Key Takeaway: Even after INT8 quantization, Ammu-1.0-Tiny-32M-ONNX-INT8 outperforms DeepSeek V4 Flash 0731 by +5.08% and GPT-6-luna by +7.91% overall Macro-F1 points. The margin is particularly striking on Medium (+9.89% over DeepSeek, +14.14% over GPT-6) and Hard boundary queries (+5.38% over DeepSeek, +7.68% over GPT-6), providing low-latency enterprise guardrailing at 31.50 ms CPU latency and zero token cost.

Quantization Technical Specifications

• Quantization Framework: Microsoft ONNX Runtime Quantization Tools (onnxruntime.quantization) via Hugging Face Optimum.
• Quantization Approach: Dynamic INT8 Post-Training Quantization with Sensitive Layer & Head FP32 Preservation.
• Quantized Operators: MatMul, Gemm (attention projections, feed-forward linear layers outside sensitive layers).
• Excluded Operators: Data-driven sensitive intermediate transformer layers and classification head nodes (classifier, pooler, head, out_proj) preserved in full FP32.
• Weight Precision: Signed 8-bit Integer (QuantType.QInt8), per-tensor scaling.
• Activation Precision: Dynamic INT8 (quantized on-the-fly per request on CPU registers).
• Calibration Requirement: None (Dynamic PTQ requires zero calibration data).
• Target Runtime Execution Provider: CPUExecutionProvider.

Limitations

• CPU Execution Target: This INT8 dynamic ONNX artifact is specifically tailored for CPU inference via ONNX Runtime. For GPU acceleration, native TensorRT FP16/INT8 execution engines are recommended.

• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.

• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-off-on-topic-guardrail-tiny-32m-onnx-int8-2026,
  title = {Ammu-1.0-Tiny-32M-ONNX-INT8: Quantized Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-tiny-ONNX-INT8}
}