AlphaRoute-0.8B-v1.0-MLX-8bit 💎

[!IMPORTANT] A newer version of AlphaRoute is available!
Check out AlphaRoute-0.8B-v1.5, the latest iteration of the AlphaRoute family of models.
Key v1.5 Highlights:

• Banking77 Intent: 95.40% (+3.60% over v1.0)
• CLINC150 Multi-Domain: 94.00% (+18.80% over v1.0)
• HWU64 Zero-Shot: 89.96% (+6.04% over v1.0)
• Golden Enterprise 300: 97.67% (+8.34% over v1.0)
• Adversarial Out-of-Scope Rejection: 100.00% (vs 35.0% on v1.0)
• Fully mitigated false-rejection bias and supports advanced multi-target schema extraction.

AlphaRoute-0.8B-v1.0 is a compact language model specialized for instruction-conditioned semantic intent routing and structured information extraction. Given a user query and a dynamically defined set of categories with semantic descriptions, it predicts the most appropriate intent, extracts parameter slots, and produces a constrained structured JSON decision.

Unlike traditional classifiers bound to fixed output heads, AlphaRoute operates as a zero-shot meta-router: you can define arbitrary runtime categories, out-of-scope policies, parameter extraction slots, and target nested JSON schemas on the fly without retraining or fine-tuning the model.

This repository provides the 8-bit high-precision MLX format, optimized for maximum accuracy and fidelity on Apple Silicon Metal GPUs (M1/M2/M3/M4).

---

📊 Benchmark Highlights

Evaluated across standardized held-out public benchmark test splits:

Note on Benchmarks: Benchmark scores reported above are evaluated on the official test splits using the unquantized PyTorch BF16 reference model. Quantized variants (MLX 8-bit/4-bit, GGUF Q8_0/Q4_K_M) provide ultra-compact memory footprints and accelerated inference.

---

🧠 Advanced Capabilities Beyond Simple Routing

While traditional classifiers only output a flat integer class ID, AlphaRoute-0.8B is trained as a structured semantic engine capable of rich schema synthesis and contextual extraction:

1. 🏗️ Arbitrary Deeply Nested JSON Schemas

AlphaRoute natively follows complex multi-level JSON schemas, including nested metadata blocks, arrays of action items, and multi-tier operational telemetry:

{
  "routing": {
    "target_category": "iam_privilege_escalation",
    "severity": "CRITICAL",
    "out_of_scope": false
  },
  "extracted_entities": {
    "compromised_identity": "admin-service-account",
    "affected_resource": "arn:aws:iam::123456789:role/ProdAdmin",
    "action_taken": "policy_attachment"
  },
  "recommended_actions": [
    "revoke_active_session",
    "quarantine_role"
  ]
}

2. 🔍 Dynamic Slot Extraction & Parameter Parsing

Extracts contextual parameters (amounts, dates, currencies, affected servers, user accounts, error codes) directly into arbitrary developer-defined keys in a single forward pass without needing a secondary NER model.

3. 🛡️ Out-of-Scope (OOS) Rejection Guardrail

When user input does not match any candidate categories in the active ontology, AlphaRoute reliably flags "out_of_scope": true and sets "intent": null, preventing hallucinated routing.

4. 🧩 Zero-Shot Runtime Schema Adaptability

You can change the schema layout, field names, or category definitions on every single API request. The model conditions itself dynamically to mirror the requested developer schema.

💡 Best Practices & Prompt Engineering Tips

To achieve optimal routing accuracy and reliable JSON generation from AlphaRoute-0.8B, follow these established best practices:

1. 🏷️ Always Provide Actionable Category Descriptions

Rather than providing bare category names, supply 1–2 sentences defining the semantic boundaries and trigger conditions:

• Avoid: - billing
• Recommended: - billing: Inquiries regarding invoices, credit card charges, overbilling, refund requests, or payment disputes.

2. 📋 Specify an Explicit JSON Output Schema

Explicitly list allowed values inside the schema definition to guide token generation:

{
  "intent": "billing | technical_support | account_security | null",
  "out_of_scope": "boolean"
}

3. 🎯 Use 1–2 In-Context Few-Shot Exemplars for Complex Logic

For nuanced domains, complex entity extractions, or custom severity scoring, providing 1–2 dynamic few-shot examples dramatically steers the output:

EXAMPLES:
Input: "Someone from Russia logged into our admin console"
Output: {"intent": "account_security", "severity": "CRITICAL", "out_of_scope": false}

Input: "Can you change my invoice address?"
Output: {"intent": "billing", "severity": "LOW", "out_of_scope": false}

📐 Canonical Prompt Structure

For highest fidelity, format your prompts following this standard block structure:

You are a precise semantic routing engine. Output only valid JSON.
TASK:
<High-level routing objective>

CATEGORIES:
- <category_name_1>: <semantic definition and trigger rules>
- <category_name_2>: <semantic definition and trigger rules>

OUTPUT SCHEMA:
<JSON Schema with allowed literals>

[OPTIONAL] EXAMPLES:
Input: "<exemplar_query>"
Output: <exemplar_json>

INPUT:
"<user_query>"

JSON:

---

⚡ Quickstart (Apple MLX)

from mlx_lm import load, generate

model, tokenizer = load("NamanAgnih0tri/AlphaRoute-0.8B-v1.0-MLX-8bit")

prompt = '''You are a precise semantic routing engine. Output only valid JSON.
TASK:
Route infrastructure telemetry alert and extract parameters.
CATEGORIES:
- database_deadlock: Transaction lock contention and timeout.
- crypto_mining: Unauthorized high CPU/GPU compute spike.
OUTPUT SCHEMA:
{"routing": {"target_category": "database_deadlock | crypto_mining | null", "severity": "CRITICAL | HIGH | LOW", "out_of_scope": "boolean"}, "telemetry": {"affected_cluster": "string or null", "metric": "string or null"}}
INPUT:
"PostgreSQL cluster 'orders-db' reports 180 active sessions blocked waiting on table lock."
JSON:'''

response = generate(model, tokenizer, prompt=prompt, max_tokens=100, verbose=False)
print(response)

---

📦 Model Specs

• Parameters: 0.8 Billion (Qwen architecture backbone)
• Quantization: Native MLX 8-bit (8.5 bits/weight)
• Disk Footprint: 763 MB
• Context Window: 2,048 tokens
• Target Hardware: Apple Silicon Mac (M1/M2/M3/M4)