Qwen3.5-4B-Router-1M
A 4B open decider for one job: what should this agent do next, and which tool should it call? It returns a calibrated probability for each option, not free text.
Qwen3.5-4B + an Unsloth decision head (LoRA r=16), trained on 17,600 rows with source labels only. No Jev answers were used in training.
Results
Scored on the held-out test split, 480 rows from 4 task families. Jev 1.13's answers (TypeSafe) are the ones stored with those rows. None of these rows shares a group or a state with the 976,219 train and validation rows of the merged 1M set.
| Task | Rows | This model | Jev 1.13 | Always the most common answer |
|---|---|---|---|---|
| Next action, agent | 120 | 85.0% | 74.2% | 50.0% |
| Next action, customer service | 120 | 83.3% | 75.8% | 50.0% |
| Tool choice, agent | 120 | 90.8% | 82.5% | 14.2% |
| Tool choice, customer service | 120 | 86.7% | 90.0% | 23.3% |
| All 4 | 480 | 86.5% | 80.6% |
Difference: +5.8 points, 95% interval +1.7 to +10.1 (bootstrap over 364 groups). This model alone was right on 58 rows, Jev alone on 30.
Limits
- Not a clinical model. Training and test rows are customer-service and software-agent traces. It has not been tested on clinical agents and must not decide anything about a patient.
- Home advantage. The labels are what the source trace did next or a tool-call parse, not a human verdict. This model learned from those same labels and Jev did not, so read the table as agreement with the source.
- Not a trace judge. On
trace_dispositionit gave one answer on all 120 rows, 34.2% against a majority class of 43.3%.trace_outcomeis 47.5% against 41.7%. Don't use it for those. - Long rows untested. The test covers states up to 13,000 characters, about 70% of these families' test rows, and the model reads at most 4,096 tokens. Jev scores a little higher on the longer rows.
- Five more trained families had a single class in this test, so their 100% says nothing and is left out.
- Training stopped on a time budget at 0.67 of an epoch, on one A100 in 1.9 hours.
Use
from unsloth import FastDecisionModel
model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Router-1M", max_seq_length=4096, load_in_4bit=False)
FastDecisionModel.for_inference(model)
question = {"decision": {"type": "choice",
"instructions": "Which available tool should be called next?",
"criteria": {"get_order_status": "Call `get_order_status`.",
"transfer_to_human_agent": "Call `transfer_to_human_agent`."}}}
state = '{"available_tools": [{"name": "get_order_status"}, {"name": "transfer_to_human_agent"}], '\
'"conversation": [{"role": "user", "content": "I would like to speak to a person please."}]}'
print(FastDecisionModel.predict(model, tokenizer, state, question))
Data
Trained on the source labels of four public Jev datasets. Jev's answers in those datasets were not used in training.
- Jev-Agent-Tool-Decisions-180K
- Jev-Customer-Support-Decisions-130K
- Jev-Agent-Trace-Audit-100K
- Jev-Intent-Domain-Routing-130K
Credit
- Base model: Qwen3.5-4B by the Qwen team, Apache-2.0. Recipe: Unsloth
FastDecisionModel. - The rows behind those datasets come from NVIDIA's Nemotron-SFT-Agentic-v2 and Open-SWE-Traces, both CC BY 4.0. They were turned into decision questions with one label each. See those datasets for their terms.