Soro-Kev-4B

pip install soro-boro

Model summary

Soro-Kev-4B is a Single-Prefill Causal Listwise Tool Reranker & Multi-Task Pointer Classifier with an Explicit __abstain__ Slot (1 forward pass for menus $\le 16$ tools; 2-stage tournament chunking for larger menus). It reads the state (user prompt, conversational history, and registered tool schemas) and produces a closed-set probability distribution over the options supplied with a question. Each question is scored in a prefill-only forward pass; the model does not generate text.

It is built directly on top of Jared Palmer's Kev pointer architecture, addressing an explicit limitation documented in the original Kev model cards ("Tool-call routing: its When2Call accuracy is below chance (0.133). Do not use it for tool-call routing"). Soro-Kev-4B is fine-tuned specifically on the multi-protocol Universal Tool Harness, raising tool routing accuracy to 90.58% (1,087/1,200) on held-out MCP & agent benchmarks, and 91.00% (91/100) on BFCL v3 tool selection.

It consists of a LoRA adapter ($r=16, \alpha=32$) and a pointer head on Qwen/Qwen3-4B. In 4-bit NF4 quantization, it operates in ~4,048 MB VRAM with a median prefill latency of ~648.6 ms on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe Gen 4, PyTorch 2.6.0, CUDA 12.4). It is intended for production AI agents, complex Model Context Protocol (MCP) tool catalogs (50+ tools), and backend API routing engines where generative LLM JSON hallucinations and latency overhead cannot be tolerated.

---

Model details

---

Input and Question Types

Soro-Kev-4B evaluates a state and any number of named questions across three typed categories:

Each question is scored by its own forward pass in the current inference code. Questions do not share a cached state prefix, and the model does not promise a fixed number of questions per forward.

---

Intended Uses

• Production AI Agent Tool Routing: Selecting tools from catalogs of 50+ schemas through a shortlist/chunking path without token-by-token JSON generation.
• Model Context Protocol (MCP) Server Gateways: High-accuracy routing across heterogeneous tool protocols (MCP JSON, REST APIs, GraphQL, Python callables).
• Calibration-Aware Action Boundary: Production routing natively applies Split Conformal LAC prediction sets ($S = { i \mid p_i \ge 1 - \hat{q} }$) loaded directly from the checkpoint's temperature.json ($\hat{q} = 0.8066$).
• Out-of-Domain Rejection: Calibrated abstention (75.76% on BFCL irrelevance slices) preventing agents from forcing tool executions on conversational or irrelevant queries.
• Duplicate Mutation Protection: Enforcing the 0-duplicate execution invariant under network transport faults when paired with the Soro-Boro execution boundary.

Out-of-Scope Uses

• Text generation, conversation, or creative writing: The model runs in prefill-only mode and has no generative token decode loop.
• Open-ended text synthesis: The model selects and normalizes schemas; it does not author free-form documents.
• Fully autonomous execution of high-risk actions: Critical financial or destructive operations should always require operator confirmation (ASK / human-in-the-loop).
• Languages other than English: Non-English tool descriptions and prompts are not validated.

---

How to Use

1. High-Level Python SDK (Soro Harness)

from soro.harness.pipeline import SoroHarness

# Loads directly from Hugging Face in NF4 quantization (4 GB VRAM)
harness = SoroHarness.from_pretrained("nisaralll/soro-kev-4b")

@harness.tool(description="Look up customer order details", side_effect="read_only")
def get_order_status(order_id: str) -> dict:
    return {"order_id": order_id, "status": "shipped", "tracking": "TRK-9810"}

@harness.tool(description="Cancel a warehouse shipment", side_effect="mutation")
def cancel_shipment(order_id: str) -> dict:
    return {"order_id": order_id, "cancelled": True}

# Fast Calibrated Routing
decision = harness.route("Where is my package for order ORD-1092?")
print("Decision Action:", decision.action)          # "EXECUTE"
print("Selected Tool:", decision.selected_tool)     # "get_order_status"
print("Confidence:", decision.probabilities.get("get_order_status"))

# Execute with Automatic Argument Coercion
result = harness.execute("get_order_status", {"order_id": "ORD-1092"})
print("Execution Result:", result.data)

2. Low-Level In-Process Scorer

from soro.routing.kev_local import LocalKevScorer

scorer = LocalKevScorer(run="nisaralll/soro-kev-4b", backend="neural")
scorer.load()

state = "User Request: Check available disk storage in /var/log"
questions = {
    "tool_route": {
        "type": "choice",
        "instructions": "Which tool should execute for this request?",
        "criteria": {
            "get_disk_usage": "Return total, used, and available disk storage for path.",
            "list_processes": "List active system processes and PIDs.",
            "__abstain__": "None of the listed tools fit."
        }
    }
}

result = scorer.evaluate_turn(state, questions)
print(result["answers"]["tool_route"])  # "get_disk_usage"

---

Training Data & Target Reconciliation

Soro-Kev-4B was trained on the nisaralll/soro-universal-tool-harness dataset (train.jsonl), consisting of exactly 23,192 multi-task state rows. Each row contains five question targets. During training, the non-permuted questions are packed into one sequence and the permutation probe is a second sequence in the same batch; this is a training batching strategy, not a guarantee that inference evaluates all questions in one pass. The run yields 115,960 machine-counted supervision targets:

Evaluation suites (val_external_human.jsonl with 1,200 held-out queries, and val_zeroshot.jsonl with 2,220 queries across 55 unseen packages) are strictly disjoint from training:

• Package Overlap: 0 packages (0 / 5 external, 0 / 55 zero-shot).
• Exact Query Overlap: 0 queries across all splits.
• Near-Duplicate Queries: 0 pairs at 3-gram Jaccard $\ge 0.75$, 1 pair at dense embedding cosine $\ge 0.88$.
• Tool Name Overlap: 4 generic tools (check_service_status, list_processes, reverse_geocode, transfer_to_human_agents) account for 162 rows (13.50%) in external holdout; 1,038 rows (86.50%) feature strictly disjoint tools whose names never appear in train.jsonl. Both subsets are reported separately below.
• Audit Verification: Committed in scripts/verify_dataset_disjointness.py and benchmarks/disjointness_audit.json.

---

Evaluation Results

All models and baselines were evaluated on an NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM, PCIe 4.0 x4, PyTorch 2.6.0, CUDA 12.4). The coverage and prediction-set metrics below come from the calibration/evaluation pipeline over 100 random splits (N_cal = 500, nominal coverage 1 - alpha = 0.90). They are distinct from the production top-mass gate.

1. Decision Model & Retrieval Baseline Comparison (Full 1,200 External Rows across 115 Unique Base Queries)

Note on Query Stratification: The 1,200 evaluations comprise 115 unique base queries tested across ~10 format/distractor views (1,095 reference MCP rows + 105 $\tau^2$ persona-dialogue rows). Soro-Kev-4B achieves 96.89% (1,061/1,095) on reference MCP tools and 24.76% (26/105) on $\tau^2$, where 64.56% of misses are calibrated abstentions or preliminary status checks.

To investigate why classical retrievers fail on tool catalogs, all retrievers were evaluated with clean schema text parsing (unwrapping JSON/REST docstrings) and calibrated with an explicit abstention threshold $\tau_{abstain}$ tuned on calibration splits:

*Note on Abstention: The 135 abstain rows represent 15 unique base queries evaluated across 9 format/distractor views each ($15 \times 9 = 135$). Denominators strictly reconcile: $1,065 + 135 = 1,200$. Numerators strictly reconcile: e.g. 0.8B gets $926 + 102 = 1,028$; 4B gets $964 + 123 = 1,087$. Dedicated adversarial refusal is evaluated on the 33-row BFCL v3 Irrelevance benchmark (Table 5).

Pointer Network Guarantee: The dual pointer head restricts selection strictly to candidate keys in context + __abstain__. It generates 0% invalid or invented tool names by construction. True misclassification rate ($1 - \text{Top-1 Accuracy}$) is 9.42% (113/1200) on external holdouts.

2. Why Classical Retrievers Fail on Tool Catalogs (Failure Analysis)

Hand-review of 20 failure cases reveals three systematic root causes for classical retriever weakness:

1. Directional Semantic Inversion: Queries like "Give me the street address" require reverse_geocode (takes coordinates $\to$ outputs address). geocode_location (takes address $\to$ outputs coordinates) contains the keyword "address" in its input parameters, causing BM25 and Dense retrievers to pick the wrong direction.
2. Lexical Synonym Gaps: User prompts like "What is the current RAM consumption?" or "Change the file type of this image" have 0 token overlap with tools named get_memory_usage ("Returns memory...") or convert_image_format ("Converts image format..."). Keyword matching scores 0.0 and falls back to incorrect tools or premature abstention.
3. Prompt Instruction Noise: Real user dialogue prompts (e.g. from Tau-2) include conversational instructions ("If the agent suggests actions that don't fix it, express mild frustration..."). Incidental words like "suggest", "read", or "plan" trigger false matches against unrelated tools like read_environment_variable or change_plan.

3. Per-Package Accuracy Table (External Holdout Breakdown)

*Note on Tau-2 persona simulation: Across the 1,095 reference MCP tools (system, geo, media), Soro-Kev-4B achieves 96.89% (1,061/1,095). On the 105 Tau-2 multi-turn dialogue instructions, Soro-Kev correctly refuses to blindly fire backend mutation tools without customer identifiers: 64.56% (51/79) of Soro-Kev-4B's misses were explicit abstentions (21 rows) or preliminary read/status lookups (30 rows, e.g. check_service_status, get_device_details) rather than premature account mutations.

4. Unseen Tool Package Split (2,220 evaluations across 55 held-out packages)

Note on Queries & Schemas: Queries share structural template distributions with training while all 55 tool schemas and packages are strictly disjoint (0 package overlap with the 319 training packages).

5. Berkeley Function-Calling Leaderboard (BFCL v3 Diagnostic, 100 rows)

*Hardware & Latency Note (RTX 4050 6GB Laptop GPU, Windows WDDM):

• Architecture Difference (Prefill vs. Autoregressive Decode): Soro-Kev outputs routing + key-mapping decisions scoring candidate tools and an explicit __abstain__ anchor via a dual linear pointer head in 1 single prefill forward pass (124.2 ms for 0.8B, 648.6 ms for 4B) with 0 generated decode tokens. In contrast, generative models (xLAM-2-1b-fc-r, Hammer2.1-1.5b) generate full JSON tool calls with argument values (~27–32 decode tokens), requiring 1 prefill pass plus 24-32 sequential autoregressive decode steps (one forward pass per token).
• In-VRAM vs. Sustained / Paged Latency: In clean isolated VRAM bursts, Arch-Router-1.5B achieves a median latency of 1,212.9 ms (~24 tokens generated). xLAM-2-1b-fc-r (mean prompt 406 tokens, mean gen 27 tokens) achieves 7,106.9 ms median in clean VRAM (initial queries: ~1,458-2,549 ms; laptop GPU thermal throttling at 85C settles to ~7.1s). Hammer2.1-1.5b (generating full markdown JSON arrays of ~32 tokens) settles to 10,564.5 ms median. When multiple generative models were evaluated sequentially without intermediate process garbage collection, Windows WDDM silently paged KV caches into Shared GPU Memory across PCIe, increasing latencies to ~10.4s-11.5s.
• Dense Cosine & Neural Baselines: All 100 BFCL rows for Dense Cosine were evaluated directly using SentenceTransformer / all-MiniLM-L6-v2 forward passes on CUDA timed with time.perf_counter(), yielding a genuine hardware median latency of 8.5 ms (6.3-60.4 ms range, 64/100 correct).

6. Accuracy by Schema Format (Independently Sourced Holdout)

• Agent Skills (SKILL.md): 100.0% (29/29)
• GraphQL RPC: 96.8% (30/31)
• REST / OpenAPI: 96.2% (25/26)
• Mixed / Heterogeneous Catalogs: 94.6% (35/37)
• MCP JSON-RPC: 94.4% (34/36)
• Python Signatures: 87.8% (36/41)

---

Architectural Trade-Offs & v0.3 Roadmap

1. Left-to-Right Causal Attention Asymmetry

In autoregressive backbones (Qwen/Qwen3-4B), attention is strictly causal (unidirectional). When tools are serialized in a single sequence: $$\text{[Query]} \dots \text{[Tool 1]} \dots \text{[Tool 2]} \dots \text{[Tool } N\text{]} \dots \text{[abstain]}$$ Tokens for Tool $N$ attend to the query and all preceding tools ($1 \dots N-1$), but Tool 1 cannot attend to subsequent tools. While training includes an auxiliary permutation consistency term (JSD) to enforce position invariance, causal asymmetry induces an empirical 1.5%–3.0% order-flip rate under arbitrary candidate permutations on closely matched tool boundaries.

2. Context & Latency Scaling

Causal prefill complexity scales with the total token length of serialized tool schemas. While compact schemas fit comfortably within context (16 tools $\approx$ 1,200 tokens), scaling to 100+ schemas requires two-stage tournament chunking or lexical pre-filtering.

3. v0.3 Architectural Direction: Bidirectional Distillation + Set Transformer

To eliminate causal order sensitivity and achieve true $O(1)$ permutation invariance, the v0.3 development branch focuses on:

1. Bidirectional Encoder Backbone: Distilling Soro-Kev-4B knowledge into a compact 149M bidirectional cross-encoder (e.g. ModernBERT or DeBERTa-v3).
2. Permutation-Invariant Set Transformer Head: Replacing causal span projections with an Induced Set Attention Block (ISAB) operating over pooled candidate tool representations, guaranteeing mathematical permutation invariance ($f(\pi(T)) = \pi(f(T))$).
3. Sub-15ms Edge Inference: Combining INT8/ONNX runtime quantization with bidirectional pooling for single-digit millisecond routing on commodity CPU and edge hardware.

---

Provenance and Reproducibility

• Repository: github.com/nisaral/Soro-Boro (release tag v0.2.1)
• Trained Checkpoint: Adapter weights and head.pt with learned temperature 0.070878 and calibration quantile q_hat = 0.8066
• Harness Dataset: huggingface.co/datasets/nisaralll/soro-universal-tool-harness
• Disjointness Audit: Committed in benchmarks/disjointness_audit.json

---

Citations

If you use Soro-Kev or Universal Tool Harness in your research or production systems, please cite the following publications:

@software{nisar2026sorokev,
  author = {Nisar, Keyush},
  title = {Soro-Kev: Calibration-Aware System-One Tool Router and Execution Boundary},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/nisaral/Soro-Boro}},
  note = {Release v0.2.1}
}

@misc{palmer2026kev08b,
  author = {Palmer, Jared},
  title = {Kev-0.8B: a calibrated decision model on Qwen3.5-0.8B},
  year = {2026},
  howpublished = {\url{https://huggingface.co/jaredpalmer/kev-0.8b}},
  note = {Kev 1.0}
}

@inproceedings{shi2025toolret,
  author = {Shi, Zhengliang and Gao, Shen and Chen, Xiuying and Wang, Lingzhi and Pei, Jiahuan and Guan, Xinyu and Ren, Zhaochun},
  title = {Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  pages = {11624--11639},
  year = {2025},
  url = {https://aclanthology.org/2025.findings-acl.647/}
}

@inproceedings{mishra2026mcpsemantic,
  author = {Mishra, Amritansh and Panditrao, Yash and Parab, Shlok and Sharma, Achintya and Mishra, Payal and Saha, Ishani},
  title = {Scalable Semantic Routing and QoS-Aware Provider Selection for MCP Agents},
  booktitle = {Procedia Computer Science, International Conference on Machine Learning and Data Engineering (ICMLDE 2026)},
  year = {2026},
  howpublished = {\url{https://github.com/amrittmishra/mcp-semantic-tool-routing-benchmark}}
}

@misc{mo2025livemcpbench,
  author = {Mo, Guozhao and others},
  title = {LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?},
  howpublished = {Preprint},
  year = {2025}
}

@inproceedings{ross2025when2call,
  author = {Ross, Hayley and Mahabaleshwarkar, Ameya Sunil and Suhara, Yoshi},
  title = {When2Call: When (not) to Call Tools},
  booktitle = {Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
  year = {2025}
}