Ammu-1.0-Base-150M-ONNX-INT8

Quantized Off On Topic Guardrail Classifier | ModernBERT Architecture | 150M Parameters (INT8 Dynamic PTQ) | 89.64% Macro-F1

Overview

Ammu-1.0-Base-150M-ONNX-INT8 is the Pareto-optimal post-training quantized (PTQ) ONNX INT8 variant of Ammu-1.0-OffOnTopic-Guardrail-base, a cross-encoder guardrail model trained to determine whether an incoming user query is on-topic (On Topic) or off-topic (Off Topic) relative to an assistant's system prompt. It is trained on a diverse, high-quality curated dataset of (system_prompt, user_prompt) pairs spanning 336 specialized domains across 40 major industries.

Generated using Dynamic INT8 Post-Training Quantization (PTQ) with Selective Classification Head Preservation, this model compresses the full 150M ModernBERT FP32 graph from 571.3 MB down to 256.0 MB (55.2% storage reduction / 2.23x compression) while slashing single-query P50 CPU latency from 159.80 ms to 95.81 ms (1.67x CPU speedup) and lifting throughput to 7.5 queries/sec (+56.3%).

Across the curated holdout benchmark of 4,008 test instances, the quantized model retains 98.0% of its FP32 baseline capability, achieving an overall Macro-F1 score of 89.64% and a Hard Macro-F1 score of 89.24%. It outperforms frontier LLMs including DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%) by +7.69% and +10.52% Macro-F1 points respectively, with zero API token cost and low CPU memory requirements.

Key Features

• High Quantized Macro-F1 Performance: Achieves an 89.64% overall Macro-F1 score and 89.24% Hard Macro-F1 score, retaining 98.0% of the FP32 baseline performance and outperforming frontier LLMs like DeepSeek V4 Flash 0731 (81.95%) and GPT-6-luna (79.12%).

• 55.2% Reduced Memory Footprint: Compresses the model from 571.3 MB down to 256.0 MB (a 2.23x compression ratio), substantially lowering memory overheads.

• 1.67x CPU Latency Speedup: Slashes single-query P50 latency from 159.80 ms (FP32 baseline) down to sub-100ms (95.81 ms) while boosting throughput from 4.8 to 7.5 queries/sec (+56.3%) on standard CPU environments.

• Selective Classification Head Preservation: Quantizes backbone MatMul and Gemm linear projections to signed INT8 (QuantType.QInt8) while selectively retaining the classification head in full FP32 to prevent decision boundary distortion.

Usage

# Install required dependencies & remove conflicting vision/diffusion packages (For Google Colab)
!pip install uv
!pip uninstall -y torchvision diffusers > /dev/null 2>&1 || true
!uv pip install --system -U "numpy>=2.1.0" "transformers>=4.48.0" "datasets>=3.0.0" "huggingface_hub>=0.28.0" "accelerate" "tabulate" "psutil" "optimum[onnxruntime]>=1.23.0" "onnx>=1.16.0" "onnxruntime>=1.19.0" "scikit-learn"

import sys
# Prevent incompatible diffusers from breaking Optimum's ONNX runtime loader
try:
    import diffusers
except Exception:
    sys.modules["diffusers"] = None

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

# Load the quantized ONNX INT8 guardrail model and tokenizer
model_id = "kalyan-ks/Ammu-1.0-Base-150M-ONNX-INT8"

model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    file_name="model_quantized.onnx",
    provider="CPUExecutionProvider"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Initialize the Hugging Face text-classification pipeline
classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer
)

# Example system prompt defining a dedicated Python programming assistant
system_prompt = (
    "You are an expert Python programming assistant. Your task is to help developers "
    "write, optimize, and debug Python 3 code, explain standard library modules, and review "
    "Python algorithms. Do provide clean code snippets, explain exceptions, and discuss "
    "Python best practices (PEP 8). Don't write code in other languages such as JavaScript, "
    "C++, or Java. Don't answer questions on non-programming topics such as sports, general "
    "cooking, financial investing, or medical advice. Strictly decline any request outside Python programming."
)

# Format input pairs: {'text': system_prompt, 'text_pair': user_prompt}
queries = [
    {
        "text": system_prompt,
        "text_pair": "How do I implement a custom async context manager using contextlib in Python 3.12?"
    },
    {
        "text": system_prompt,
        "text_pair": "Why is my multiprocessing Pool hanging when sharing a large NumPy array across workers?"
    },
    {
        "text": system_prompt,
        "text_pair": "What are the health benefits of a Mediterranean diet and how many calories are in olive oil?"
    },
    {
        "text": system_prompt,
        "text_pair": "I'm building a Python accounting tool. Can you advise me on the best tax deduction strategies for small businesses?"
    },
]

# Run guardrail classification inference (dynamic padding enabled)
results = classifier(queries, max_length=512, truncation=True, padding=True)

# Display predictions
for query, res in zip(queries, results):
    print(f"User Prompt: {query['text_pair']}")
    print(f"Prediction : {res['label']} (Confidence: {res['score']:.4f})\n")

Model Evaluation

The quantized model was evaluated on the curated holdout benchmark test set of 4,008 balanced instances from kalyan-ks/off-on-topic-guardrail across three difficulty tiers (Easy: 635, Medium: 1,571, and Hard: 1,802). All benchmarks were executed sequentially with ONNX Runtime on CPU with unified sequence limits (max_length=512).

Post-Training Quantization (PTQ) Progression: Ammu 1.0 Base (150M)

The complete benchmark progression across different post-training quantization techniques reveals the performance and efficiency tradeoffs for the Base 150M architecture:

Key Architectural Takeaways:

1. Why Selective Head FP32 Preservation is Pareto-Optimal: The classification head projects pooled representations into final logits. Quantizing these final projection layers introduces roundoff noise directly onto output margins, degrading separation on subtle boundary queries. Retaining the head in full FP32 safeguards decision boundaries with negligible compute and memory overhead, as the vast majority of parameters and latency reside in the quantized transformer backbone.

2. Dynamic vs. Static INT8 on CPU: Static calibration approaches suffered from activation clipping degradation (-10.63% Macro-F1 drop) and higher CPU latency due to QDQ operator dequantize/quantize tensor overhead on variable-length textual sequences. Dynamic INT8 avoids activation clipping entirely.

3. Per-Channel Instability in ModernBERT: Per-channel weight quantization caused severe representation collapse (48.91% Macro-F1), whereas per-tensor dynamic quantization remained exceptionally stable.

Ammu 1.0 Guardrail Quantized Family Leaderboard

Comparison of Pareto-optimal quantized models across the four Ammu 1.0 architectural tiers:

Comparison with Frontier LLMs and FP32 Baseline

Frontier LLMs were evaluated zero-shot with structured Pydantic outputs and reasoning disabled to benchmark direct domain boundary classification capability:

Key Takeaway: Even after INT8 quantization, Ammu-1.0-Base-150M-ONNX-INT8 outperforms DeepSeek V4 Flash 0731 by +7.69% and GPT-6-luna by +10.52% overall Macro-F1 points. The margin is particularly striking on Medium (+12.20% over DeepSeek, +16.45% over GPT-6) and Hard boundary queries (+7.82% over DeepSeek, +10.12% over GPT-6), providing strict enterprise guardrailing at a fraction of the latency and zero token cost.

Quantization Technical Specifications

• Quantization Framework: Microsoft ONNX Runtime Quantization Tools (onnxruntime.quantization) via Hugging Face Optimum.
• Quantization Approach: Dynamic INT8 Post-Training Quantization with Selective Head FP32 Preservation.
• Quantized Operators: MatMul, Gemm (attention projections, feed-forward linear layers).
• Excluded Operators: Classification head nodes (classifier, pooler, head, out_proj) preserved in full FP32.
• Weight Precision: Signed 8-bit Integer (QuantType.QInt8), per-tensor scaling.
• Activation Precision: Dynamic INT8 (quantized on-the-fly per request on CPU registers).
• Calibration Requirement: None (Dynamic PTQ requires zero calibration data).
• Target Runtime Execution Provider: CPUExecutionProvider.

Limitations

• CPU Execution Target: This INT8 dynamic ONNX artifact is specifically tailored for CPU inference via ONNX Runtime. For GPU acceleration, native TensorRT FP16/INT8 execution engines are recommended.

• Language Scope: The model is trained and evaluated predominantly on English language text; classification performance on multilingual or code-switched inputs may be lower.

• Complex Mixed Intent: Prompts that deliberately weave permitted and restricted topics into a single complex composite request may present borderline confidence scores and benefit from threshold calibration.

Citation

@misc{ammu-1.0-off-on-topic-guardrail-base-150m-onnx-int8-2026,
  title = {Ammu-1.0-Base-150M-ONNX-INT8: Quantized Off On Topic Guardrail Classification Model},
  author = {Kalyan KS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/kalyan-ks/Ammu-1.0-OffOnTopic-Guardrail-base-ONNX-INT8}
}