AlphaRoute-0.8B-v1.0-MLX-4bit ⚡

[!IMPORTANT] A newer version of AlphaRoute is available!
Check out AlphaRoute-0.8B-v1.5, the latest iteration of the AlphaRoute family of models.
Key v1.5 Highlights:

• Banking77 Intent: 95.40% (+3.60% over v1.0)
• CLINC150 Multi-Domain: 94.00% (+18.80% over v1.0)
• HWU64 Zero-Shot: 89.96% (+6.04% over v1.0)
• Golden Enterprise 300: 97.67% (+8.34% over v1.0)
• Adversarial Out-of-Scope Rejection: 100.00% (vs 35.0% on v1.0)
• Fully mitigated false-rejection bias and supports advanced multi-target schema extraction.

AlphaRoute-0.8B-v1.0 is a compact language model specialized for instruction-conditioned semantic intent routing and structured information extraction. Given a user query and a dynamically defined set of categories with semantic descriptions, it predicts the most appropriate intent, extracts parameter slots, and produces a constrained structured JSON decision.

Unlike traditional classifiers bound to fixed output heads, AlphaRoute operates as a zero-shot meta-router: you can define arbitrary runtime categories, out-of-scope policies, parameter extraction slots, and target nested JSON schemas on the fly without retraining or fine-tuning the model.

This repository provides the 4-bit quantized MLX format, optimized natively for Apple Silicon Metal GPUs (M1/M2/M3/M4) with an ultra-lightweight 404 MB disk footprint and sub-300ms latency.

---

📊 Benchmark Highlights

Evaluated across standardized held-out public benchmark test splits:

Note on Benchmarks: Benchmark scores reported above are evaluated on the official test splits using the unquantized PyTorch BF16 reference model. Quantized variants (MLX 8-bit/4-bit, GGUF Q8_0/Q4_K_M) provide ultra-compact memory footprints and accelerated inference.

---

🧠 Advanced Capabilities Beyond Simple Routing

While traditional classifiers only output a flat integer class ID, AlphaRoute-0.8B is trained as a structured semantic engine capable of rich schema synthesis and contextual extraction:

1. 🏗️ Arbitrary Deeply Nested JSON Schemas

AlphaRoute natively follows complex multi-level JSON schemas, including nested metadata blocks, arrays of action items, and multi-tier operational telemetry:

{
  "routing": {
    "target_category": "iam_privilege_escalation",
    "severity": "CRITICAL",
    "out_of_scope": false
  },
  "extracted_entities": {
    "compromised_identity": "admin-service-account",
    "affected_resource": "arn:aws:iam::123456789:role/ProdAdmin",
    "action_taken": "policy_attachment"
  },
  "recommended_actions": [
    "revoke_active_session",
    "quarantine_role"
  ]
}

2. 🔍 Dynamic Slot Extraction & Parameter Parsing

Extracts contextual parameters (amounts, dates, currencies, affected servers, user accounts, error codes) directly into arbitrary developer-defined keys in a single forward pass without needing a secondary NER model.

3. 🛡️ Out-of-Scope (OOS) Rejection Guardrail

When user input does not match any candidate categories in the active ontology, AlphaRoute reliably flags "out_of_scope": true and sets "intent": null, preventing hallucinated routing.

4. 🧩 Zero-Shot Runtime Schema Adaptability

You can change the schema layout, field names, or category definitions on every single API request. The model conditions itself dynamically to mirror the requested developer schema.

💡 Best Practices & Prompt Engineering Tips

To achieve optimal routing accuracy and reliable JSON generation from AlphaRoute-0.8B, follow these established best practices:

1. 🏷️ Always Provide Actionable Category Descriptions

Rather than providing bare category names, supply 1–2 sentences defining the semantic boundaries and trigger conditions:

• Avoid: - billing
• Recommended: - billing: Inquiries regarding invoices, credit card charges, overbilling, refund requests, or payment disputes.

2. 📋 Specify an Explicit JSON Output Schema

Explicitly list allowed values inside the schema definition to guide token generation:

{
  "intent": "billing | technical_support | account_security | null",
  "out_of_scope": "boolean"
}

3. 🎯 Use 1–2 In-Context Few-Shot Exemplars for Complex Logic

For nuanced domains, complex entity extractions, or custom severity scoring, providing 1–2 dynamic few-shot examples dramatically steers the output:

EXAMPLES:
Input: "Someone from Russia logged into our admin console"
Output: {"intent": "account_security", "severity": "CRITICAL", "out_of_scope": false}

Input: "Can you change my invoice address?"
Output: {"intent": "billing", "severity": "LOW", "out_of_scope": false}

📐 Canonical Prompt Structure

For highest fidelity, format your prompts following this standard block structure:

You are a precise semantic routing engine. Output only valid JSON.
TASK:
<High-level routing objective>

CATEGORIES:
- <category_name_1>: <semantic definition and trigger rules>
- <category_name_2>: <semantic definition and trigger rules>

OUTPUT SCHEMA:
<JSON Schema with allowed literals>

[OPTIONAL] EXAMPLES:
Input: "<exemplar_query>"
Output: <exemplar_json>

INPUT:
"<user_query>"

JSON:

---

⚡ Quickstart (Apple MLX)

1. Install MLX

pip install mlx-lm

2. Run Nested Semantic Extraction & Prediction

from mlx_lm import load, generate

model, tokenizer = load("NamanAgnih0tri/AlphaRoute-0.8B-v1.0-MLX-4bit")

prompt = '''You are a precise semantic routing engine. Output only valid JSON.
TASK:
Route cloud infrastructure event and extract parameters.
CATEGORIES:
- finops_budget_anomaly: Unpartitioned scans, large spend surge.
- iam_privilege_escalation: Unauthorized root policy attachment.
OUTPUT SCHEMA:
{"routing": {"target_category": "finops_budget_anomaly | iam_privilege_escalation | null", "severity": "CRITICAL | HIGH | LOW", "out_of_scope": "boolean"}, "metadata": {"affected_resource": "string or null", "cost_or_metric": "string or null"}}
INPUT:
"BigQuery project 'analytics-warehouse' incurred $14,250 in 4 hours due to cross-join scans."
JSON:'''

response = generate(model, tokenizer, prompt=prompt, max_tokens=100, verbose=False)
print(response)

---

📦 Model Specs

• Parameters: 0.8 Billion (Qwen architecture backbone)
• Quantization: Native MLX 4-bit (4.5 bits/weight)
• Disk Footprint: 404 MB
• Context Window: 2,048 tokens
• Target Hardware: Apple Silicon Mac (M1/M2/M3/M4)