Decision-2.0-Eos-0.8B (ONNX)

ONNX conversion of vllm-sr/Decision-2.0-Eos-0.8B, a Decision 2.0 model from vLLM Semantic Router: give it a state and typed questions (choice, noul, score) and it returns a probability for every option. For the model, its training and evaluation, see the original card.

It runs in the browser on WebGPU with open-jev (model: "decision2-eos-0.8b"), or with ONNX Runtime anywhere.

How it was made

Decision 2.0 is a full fine-tune of Qwen3.5-0.8B with a candidate head on the final hidden states. The weights were put into the graphs of onnx-community/Qwen3.5-0.8B-ONNX-OPT (fused LinearAttention / CausalConvWithState), mapped by name with transforms found by value for the same graphs (transpose, 1 + weight for the zero-centred RMSNorm, -exp(A_log)), and quantized with ONNX Runtime's own kernels. The LM head is dropped; embeddings, decoder (empty caches and position ids built in-graph) and head are fused into one graph.

input_ids [B, L], attention_mask [B, L], answer_pos [B], option_pos [B, K]  ->  logits [B, K]

Build one prompt per question as the package's decision_model.segments / encode do (Context: … Task type: … Question: … Options:, one <option> segment per option holding {"description": …, "key": …} as sorted, compact JSON, then the Decision: suffix), tokenizing each segment separately; option_pos is the last token of each option segment and answer_pos the last token. Add decision2.score_bias[L] from config.json to the logits of an L-level score question (none for this model), then softmax (temperature 1).

Scripts and reference cases are in conversion/.

Files

q8 here is 8-bit block quantization (MatMulNBits, bits=8), not the usual dynamic int8 export; it uses the q8 slot because Transformers.js has no q8f16.

Checks

Against the packaged runtime (decision2.api.Decision2, exact path, CPU fp32), on 11 questions over 6 states: largest difference in any option's probability and how many top answers changed.

open-jev in Node reproduces the Python run of the same graph exactly (same token ids, option positions and score offsets).

In the browser

On a MacBook Pro (M3 Pro, Chrome, WebGPU), on the 400 states × 5 questions of typed-decisions (test split; top answer = annotator label, as in the open-jev demo's benchmark):

License

Apache-2.0, as the original model.