⚡ Soro-Kev-0.8B: Ultra-Fast Edge Tool Router

Soro-Kev-0.8B is an ultra-fast, sub-100ms System-One Pointer Model designed for edge devices, local AI assistants, MCP servers, and high-throughput routing pipelines.

Built on Qwen/Qwen3.5-0.8B-Base, it predicts tool selections and resolves schema arguments in a single forward prefill pass using hidden-state pointer attention, without generating text or slow JSON tokens.

User Request + Candidate Tool Schemas
            │
            ▼  [Single Forward Prefill — 80ms on Laptop GPU / Edge]
    Hidden State Pointer Scoring: h_query · h_tool / T
            │
            ▼
    Softmax-LAC Conformal Prediction Set:
    ├── |S| = 1  ──▶  EXECUTE (Immediate safe action)
    ├── |S| > 1  ──▶  ASK     (Targeted clarification)
    └── Abstain  ──▶  ABSTAIN (Safe rejection)

---

⚡ Highlights

• 🏎️ Ultra-Fast Edge Latency: ~80 ms on consumer laptop GPU (sub-30ms on Linux kernels).
• 💾 Tiny Footprint: Only 1,563 MB VRAM peak footprint. Runs effortlessly alongside small LLMs on single GPUs or edge chips.
• 🛡️ Zero Hallucination: Only points to registered tools. Cannot invent nonexistent tool names.
• 📐 Calibrated Decision Threshold: Uses split conformal prediction (calibrated_q_hat = 0.9518).
• 🔌 Universal Compatibility: Works with MCP tools, Python signatures, REST/OpenAPI, and Agent Skills (SKILL.md).

---

📦 Quickstart

from soro.harness.pipeline import SoroHarness

# Load 0.8B model directly from Hugging Face
harness = SoroHarness.from_pretrained("nisaralll/soro-kev-0.8b")

@harness.tool(description="Check account balance for customer", side_effect="read_only")
def get_balance(customer_id: str) -> dict:
    return {"customer_id": customer_id, "balance_usd": 1420.50}

@harness.tool(description="Send payment transfer to recipient", side_effect="mutation")
def transfer_funds(to_user: str, amount: float) -> dict:
    return {"to_user": to_user, "amount": amount, "status": "transferred"}

# 80ms Fast Routing
decision = harness.route("Check how much money Alice has in her account")
print("Action:", decision.action)          # "EXECUTE"
print("Tool:", decision.selected_tool)     # "get_balance"

# Automatic argument normalization & execution
res = harness.execute("get_balance", {"customerId": "usr_992"})
print("Result:", res.data)

---

📊 Benchmark Results

Evaluated on an NVIDIA GeForce RTX 4050 Laptop GPU and Kaggle T4 GPUs using seed-42 slices (100 calibration / 200 test rows).

---

📜 License

Released under the Apache-2.0 License.