Soro-Kev-0.8B

pip install soro-boro

Model summary

Soro-Kev-0.8B is a Single-Prefill Causal Listwise Tool Reranker & Multi-Task Pointer Classifier with an Explicit __abstain__ Slot (1 forward pass for menus $\le 16$ tools; 2-stage tournament chunking for larger menus). It reads the state (user prompt, conversational history, and available tool schemas) and produces a closed-set probability distribution over the options supplied with a question. Each question is scored in a prefill-only forward pass; the model does not generate text.

It is built directly on top of Jared Palmer's Kev pointer architecture, addressing an explicit limitation documented in the original Kev-0.8B model card ("Tool-call routing: its When2Call accuracy is below chance (0.133). Do not use it for tool-call routing"). Soro-Kev-0.8B is trained specifically on multi-protocol tool catalogs (Model Context Protocol, REST/OpenAPI, Agent Skills, and Python signatures), raising tool routing accuracy to 85.67% (1,028/1,200) on held-out MCP & agent benchmarks, and 83.00% (83/100) on BFCL v3 tool selection.

It consists of a LoRA adapter ($r=16, \alpha=32$) and a pointer head on Qwen/Qwen3.5-0.8B-Base. It runs in ~124.2 ms on an NVIDIA GeForce RTX 4050 Laptop GPU (1,563 MB peak VRAM, PyTorch 2.6.0). It is intended for edge deployment, local MCP desktop sidecars, and high-frequency routing pipelines where generative LLM token generation latency is prohibitive.

---

Model details

---

Input and Question Types

Soro-Kev-0.8B evaluates a state and any number of named questions across three typed categories:

Each question is scored by its own forward pass in the current inference code. Questions do not share a cached state prefix, and the model does not promise a fixed number of questions per forward.

---

Intended Uses

• AI Agent Tool Routing: Selecting tools from catalogs of 10 to 100+ schemas through a shortlist/chunking path without token-by-token JSON generation. The measured 0.8B prefill latency is about 124 ms, not a sub-100 ms guarantee.
• Model Context Protocol (MCP) Sidecars: Routing queries to local tools exposed by MCP servers over stdio or SSE.
• Calibration-Aware Action Boundary: Production routing natively applies Split Conformal LAC prediction sets ($S = { i \mid p_i \ge 1 - \hat{q} }$) loaded directly from the checkpoint's temperature.json ($\hat{q} = 0.9518$).
• On-Device & Edge Routing: Running alongside small edge LLMs on consumer laptops, Apple Silicon, or 4 GB GPUs.
• Duplicate Mutation Protection: When integrated with the Soro-Boro execution boundary, enforcing the 0-duplicate execution invariant under network dropouts.

Out-of-Scope Uses

• Text generation, conversation, or creative writing: The model runs in prefill-only mode and has no generative token decode loop.
• Arbitrary general classification without tools: Out of domain on tasks unlike its training families (e.g. sentiment classification or narrative analysis), where general models like Kev-0.8B or Kev-4B should be used.
• Fully autonomous execution of high-risk actions: Financial transfers, database drops, or administrative actions should always require operator confirmation (ASK / human-in-the-loop).
• Languages other than English: Non-English tool descriptions and prompts are not validated.

---

How to Use

1. High-Level Python SDK (Soro Harness)

from soro.harness.pipeline import SoroHarness

# Load checkpoint from Hugging Face
harness = SoroHarness.from_pretrained("nisaralll/soro-kev-0.8b")

@harness.tool(description="Check account balance for customer", side_effect="read_only")
def get_balance(customer_id: str) -> dict:
    return {"customer_id": customer_id, "balance_usd": 1420.50}

@harness.tool(description="Issue partial or full refund for settled order", side_effect="mutation")
def refund_order(order_id: str, amount: float) -> dict:
    return {"order_id": order_id, "refunded": amount}

# Fast Calibrated Routing
decision = harness.route("Refund customer $45.00 for order ORD-9921")
print(decision.action)          # "EXECUTE"
print(decision.selected_tool)   # "refund_order"
print(decision.candidates)      # ["refund_order"] (Prediction set |S| = 1)

2. Low-Level In-Process Scorer

from soro.routing.kev_local import LocalKevScorer

scorer = LocalKevScorer(run="nisaralll/soro-kev-0.8b", backend="neural")
scorer.load()

state = "User Request: Check available disk storage in /var/log"
questions = {
    "tool_route": {
        "type": "choice",
        "instructions": "Which tool should execute for this request?",
        "criteria": {
            "get_disk_usage": "Return total, used, and available disk storage for path.",
            "list_processes": "List active system processes and PIDs.",
            "__abstain__": "None of the listed tools fit."
        }
    }
}

result = scorer.evaluate_turn(state, questions)
print(result["answers"]["tool_route"])  # "get_disk_usage"
print(result["probabilities"]["tool_route"])

---

---

Training Data & Target Reconciliation

Soro-Kev-0.8B was trained on the nisaralll/soro-universal-tool-harness dataset (train.jsonl), consisting of exactly 23,192 multi-task state rows. Each row contains five question targets. During training, the non-permuted questions are packed into one sequence and the permutation probe is a second sequence in the same batch; this is a training batching strategy, not a guarantee that inference evaluates all questions in one pass. The run yields 115,960 machine-counted supervision targets:

Evaluation suites (val_external_human.jsonl with 1,200 held-out queries, and val_zeroshot.jsonl with 2,220 queries across 55 unseen packages) are strictly disjoint from training:

• Package Overlap: 0 packages (0 / 5 external, 0 / 55 zero-shot).
• Exact Query Overlap: 0 queries across all splits.
• Near-Duplicate Queries: 0 pairs at 3-gram Jaccard $\ge 0.75$, 1 pair at dense embedding cosine $\ge 0.88$.
• Tool Name Overlap: 4 generic tools (check_service_status, list_processes, reverse_geocode, transfer_to_human_agents) account for 162 rows (13.50%) in external holdout; 1,038 rows (86.50%) feature strictly disjoint tools whose names never appear in train.jsonl. Both subsets are reported separately below.
• Audit Verification: Committed in scripts/verify_dataset_disjointness.py and benchmarks/disjointness_audit.json.

---

Evaluation Results

All models and baselines were evaluated on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe 4.0 x4, PyTorch 2.6.0, CUDA 12.4). The coverage and prediction-set metrics below come from the calibration/evaluation pipeline over 100 random splits (N_cal = 500, nominal coverage 1 - alpha = 0.90). They are distinct from the production top-mass gate.

1. Decision Model & Retrieval Baseline Comparison (Full 1,200 External Rows across 115 Unique Base Queries)

Note on Query Stratification: The 1,200 evaluations comprise 115 unique base queries tested across ~10 format/distractor views (1,095 reference MCP rows + 105 $\tau^2$ persona-dialogue rows). Soro-Kev-4B achieves 96.89% (1,061/1,095) on reference MCP tools and 24.76% (26/105) on $\tau^2$, where 64.56% of misses are calibrated abstentions or preliminary status checks.

*Note on Abstention: The 135 abstain rows represent 15 unique base queries evaluated across 9 format/distractor views each ($15 \times 9 = 135$). Denominators strictly reconcile: $1,065 + 135 = 1,200$. Numerators strictly reconcile: e.g. 0.8B gets $926 + 102 = 1,028$; 4B gets $964 + 123 = 1,087$. Dedicated adversarial refusal is evaluated on the 33-row BFCL v3 Irrelevance benchmark (Section 3).

Pointer Network Guarantee: Restricts candidate selection strictly to the closed candidate set + __abstain__ (0% invalid or invented tool-name generation). The empirical misclassification rate ($1 - \text{Top-1 Accuracy}$) is 14.33% (172/1200) on external holdouts.

2. Unseen Tool Package Split (2,220 evaluations across 55 held-out packages)

Note on Queries & Schemas: Queries share structural template distributions with training while all 55 tool schemas and packages are strictly disjoint (0 package overlap with the 319 training packages).

3. Berkeley Function-Calling Leaderboard (BFCL v3 Diagnostic, 100 rows)

*Hardware & Latency Note (RTX 4050 6GB Laptop GPU, Windows WDDM):

• Architecture Difference (Prefill vs. Autoregressive Decode): Soro-Kev outputs routing + key-mapping decisions scoring candidate tools and an explicit __abstain__ anchor via a dual linear pointer head in 1 single prefill forward pass (124.2 ms for 0.8B, 648.6 ms for 4B) with 0 generated decode tokens. In contrast, generative models (xLAM-2-1b-fc-r, Hammer2.1-1.5b) generate full JSON tool calls with argument values (~27–32 decode tokens), requiring 1 prefill pass plus 24-32 sequential autoregressive decode steps (one forward pass per token).
• In-VRAM vs. Sustained / Paged Latency: In clean isolated VRAM bursts, Arch-Router-1.5B achieves a median latency of 1,212.9 ms (~24 tokens generated). xLAM-2-1b-fc-r (mean prompt 406 tokens, mean gen 27 tokens) achieves 7,106.9 ms median in clean VRAM (initial queries: ~1,458-2,549 ms; laptop GPU thermal throttling at 85C settles to ~7.1s). Hammer2.1-1.5b (generating full markdown JSON arrays of ~32 tokens) settles to 10,564.5 ms median. When multiple generative models were evaluated sequentially without intermediate process garbage collection, Windows WDDM silently paged KV caches into Shared GPU Memory across PCIe, increasing latencies to ~10.4s-11.5s.
• Dense Cosine & Neural Baselines: All 100 BFCL rows for Dense Cosine were evaluated directly using SentenceTransformer / all-MiniLM-L6-v2 forward passes on CUDA timed with time.perf_counter(), yielding a genuine hardware median latency of 8.5 ms (6.3-60.4 ms range, 64/100 correct).

---

---

Architectural Trade-Offs & v0.3 Roadmap

1. Left-to-Right Causal Attention Asymmetry

In autoregressive backbones (Qwen/Qwen3.5-0.8B), attention is strictly causal (unidirectional). When tools are serialized in a single sequence: $$\text{[Query]} \dots \text{[Tool 1]} \dots \text{[Tool 2]} \dots \text{[Tool } N\text{]} \dots \text{[abstain]}$$ Tokens for Tool $N$ attend to the query and all preceding tools ($1 \dots N-1$), but Tool 1 cannot attend to subsequent tools. While training includes an auxiliary permutation consistency term (JSD) to enforce position invariance, causal asymmetry induces an empirical 1.5%–3.0% order-flip rate under arbitrary candidate permutations on closely matched tool boundaries.

2. Context & Latency Scaling

Causal prefill complexity scales with the total token length of serialized tool schemas. While compact schemas fit comfortably within context (16 tools $\approx$ 1,200 tokens), scaling to 100+ schemas requires two-stage tournament chunking or lexical pre-filtering.

3. v0.3 Architectural Direction: Bidirectional Distillation + Set Transformer

To eliminate causal order sensitivity and achieve true $O(1)$ permutation invariance, the v0.3 development branch focuses on:

1. Bidirectional Encoder Backbone: Distilling Soro-Kev-4B knowledge into a compact 149M bidirectional cross-encoder (e.g. ModernBERT or DeBERTa-v3).
2. Permutation-Invariant Set Transformer Head: Replacing causal span projections with an Induced Set Attention Block (ISAB) operating over pooled candidate tool representations, guaranteeing mathematical permutation invariance ($f(\pi(T)) = \pi(f(T))$).
3. Sub-15ms Edge Inference: Combining INT8/ONNX runtime quantization with bidirectional pooling for single-digit millisecond routing on commodity CPU and edge hardware.

---

Provenance and Reproducibility

• Repository: github.com/nisaral/Soro-Boro (release tag v0.2.1)
• Trained Checkpoint: Adapter weights and head.pt with learned temperature 0.075487 and calibration quantile q_hat = 0.9518
• Harness Dataset: huggingface.co/datasets/nisaralll/soro-universal-tool-harness
• Disjointness Audit: Committed in benchmarks/disjointness_audit.json

---

Citations

If you use Soro-Kev or Universal Tool Harness in your research or production systems, please cite the following publications:

@software{nisar2026sorokev,
  author = {Nisar, Keyush},
  title = {Soro-Kev: Calibration-Aware System-One Tool Router and Execution Boundary},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/nisaral/Soro-Boro}},
  note = {Release v0.2.1}
}

@misc{palmer2026kev08b,
  author = {Palmer, Jared},
  title = {Kev-0.8B: a calibrated decision model on Qwen3.5-0.8B},
  year = {2026},
  howpublished = {\url{https://huggingface.co/jaredpalmer/kev-0.8b}},
  note = {Kev 1.0}
}

@inproceedings{shi2025toolret,
  author = {Shi, Zhengliang and Gao, Shen and Chen, Xiuying and Wang, Lingzhi and Pei, Jiahuan and Guan, Xinyu and Ren, Zhaochun},
  title = {Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  pages = {11624--11639},
  year = {2025},
  url = {https://aclanthology.org/2025.findings-acl.647/}
}

@inproceedings{mishra2026mcpsemantic,
  author = {Mishra, Amritansh and Panditrao, Yash and Parab, Shlok and Sharma, Achintya and Mishra, Payal and Saha, Ishani},
  title = {Scalable Semantic Routing and QoS-Aware Provider Selection for MCP Agents},
  booktitle = {Procedia Computer Science, International Conference on Machine Learning and Data Engineering (ICMLDE 2026)},
  year = {2026},
  howpublished = {\url{https://github.com/amrittmishra/mcp-semantic-tool-routing-benchmark}}
}

@misc{mo2025livemcpbench,
  author = {Mo, Guozhao and others},
  title = {LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?},
  howpublished = {Preprint},
  year = {2025}
}

@inproceedings{ross2025when2call,
  author = {Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi},
  title = {When2Call: When (not) to Call Tools},
  booktitle = {Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year = {2025}
}