Decision 4B v1.6 (FlyMyJev-4B)
An open model for typed decisions: give it a state (a ticket, a policy and a case, a log, an answer to judge, a
conversation with tools) and a question with a closed set of answers (noul yes/no, choice, score), and it returns a
probability for every declared option. Nothing is generated. Independent project: not a release of TypeSafe, not
affiliated with it, not a reconstruction of Jev's closed implementation.
v1.6 is training run qwen35_4b_letter_h19_v70: the v1.5 recipe on a broader mixture that adds many decision skills
(intent routing with many options, tool choice, claim and entailment checks, relevance, sentiment and stance, smart-home
state, chords, product search, commonsense and more). The package reads both JevBench requests and Decision Index
requests: up to 676 options per choice and inputs up to 131,072 tokens.
What it is
| Base | Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (Apache-2.0), downloaded separately |
| Adapter | LoRA rank 16, alpha 32, dropout 0.05 on the attention projections of both layer kinds (q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj); 14.4M trainable parameters |
| Prompt | the chat template with thinking disabled; one JSON user message {evidence, criterion, options[{letter, description}]}, descriptions "<key>: <text>", yes/no options true then false (SemIf's format, MIT) |
| Readout | the option letters' logits at the last prompt token (fp32 projection of its hidden state), softmax at the temperature in model.json for the request type: choice and score 1.45 (NLL on our own data only), yes/no 0.40 (kept from v1.5 so that yes/no answers stay decisive) |
| More than 26 options | read in two levels: one pass per block of up to 26 options in their given order, then one pass over the blocks' leaders; up to 26 options it is a single pass |
| Output | a distribution over exactly the declared options: a label outside that set cannot come out |
| Serving | the LoRA is folded into the base at load; one CUDA graph per padded input length; p50 18.3 ms / p95 21.5 ms per short decision on one RTX 4090 (eager 56 ms); the same answer as the eager path on 120/120 checked items |
| Inputs | up to 131,072 tokens; an input that does not fit on the GPU is reported as a capacity limit, never truncated |
Requirements
- A C compiler at run time (for example
build-essential): the flash-linear-attention Triton kernels are compiled on the first request, and CUDA-graph capture at load needs them too. model.load()verifies every file againstmanifest.jsonand refuses symlinks, so a Hugging Face cache snapshot (whose files are links intoblobs/) must be copied to a real directory first, for example withhf download --local-dir.
Use
import model # this repository's model.py
m = model.load() # verifies every file against manifest.json, downloads the pinned base on first use
m.decide("Refunds need a receipt and a purchase within 30 days. The customer bought 12 days ago and has no receipt.",
{"type": "noul", "instructions": "Is a refund permitted under the policy?"})
# {'probabilities': {'true': ..., 'false': ...}, 'input_tokens': ..., 'seconds': ...}
server.pyserves the/v1/systemonewire format: JevBench'stypesafeadapter and the Decision Index kit'shttpengine both drive it (python server.py --assets <local copy of the base> --port 8090).jevbench_adapter.pyis an in-process JevBench adapter;run_jevbench.pyruns the official runner without editing its registry. SetFLYMYJEV_DEVICE=cputo run on CPU (slow, no CUDA graphs).
Results (our measurements)
Measured on one RTX 4090 with the served code. JevBench's judge tier and sealed set are not public and are not included.
| set | frozen Qwen3.5-4B | Decision 4B v1.5 | v1.6 |
|---|---|---|---|
| JevBench public, hard (111) | 61.3 | 76.6 | 76.6 |
| JevBench public, standard (72) | 97.2 | 100.0 | 100.0 |
| JevBench public, easy (48) | 97.9 | 100.0 | 100.0 |
| our hard dev set (160, 8 families) | 42.5 | 70.6 | 71.9 |
| our real-case sets v1-v3 (407) | 88.9 | 90.7 | 91.6 |
| Decision Index 0.3, complete public run (140,178 requests) | 38.24 | 42.69 | |
| Decision Index 0.3, our fixed 2,758-request sample of the public suite | 38.99 | 43.91 |
- Decision Index: the complete public 0.3 run was made with the kit's
httpengine against this package'sserver.py, one request at a time on one RTX 4090, all 140,178 requests answered, median 31 ms; results: datasets/flymy-ai/decision-index-results, runs/decision-4b-v1.6-0.3. The sample holds 60 whole cases of each of the 37 index benchmarks. - Biggest Decision Index changes on the complete run (v1.5 -> v1.6, chance-corrected skill): chord recognition 0.04 -> 0.61, smart-home control 0.10 -> 0.28, hallucination checks 0.26 -> 0.40, select-all-that-apply 0.19 -> 0.30, sarcasm 0.18 -> 0.29, product search relevance 0.26 -> 0.36, tool-call decisions 0.51 -> 0.60, intent routing (CLINC150) 0.71 -> 0.79.
- Fidelity to exact gold distributions on the public items: 89.7 (1 - mean total variation distance).
Training
One run from the base, no merging of runs and no ensembles: one pass, 2,198 steps of at most 12,288 padded tokens (24.8M tokens), lr 3e-05 with warm-up and cosine decay, cross-entropy on the option-letter logits (the exact distribution where an item declares one), choice options shuffled on every pass, seed 99. The recipe shape follows JevK5 v0.2 (allebee/jevk5, Apache-2.0).
Data: our own decision data, licence-reviewed public datasets, and synthetic data written and checked with open-weight models. No JevBench item and no output of Jev was used for training, tuning or model selection. No Decision Index suite item was used for training or tuning; every added row was checked for word 13-gram overlap with both benchmarks. Built with Qwen: part of the tool-use data was generated with Qwen2.5-72B.
Benchmark hygiene
- The temperature was fitted on our own calibration split and hard dev set, never on benchmark items.
- The gates that made this run a release candidate were declared before the run.
- A fixed sample of the public Decision Index suite was used to choose between candidate runs, never to build or select training data.
Limitations
- One pass of a 4B model: no step-by-step reasoning. Chained arithmetic, code tracing, chess and expert-level questions stay weak.
- English only; the calibration was fitted on our own data.
Licence and credits
Adapter, code and configuration: Apache-2.0 (LICENSE). Base: Qwen3.5-4B by the Qwen team (Apache-2.0). Prompt format: SemIf by TheoLeeCJ (MIT). CUDA-graph runtime design and recipe shape: JevK5 by allebee (Apache-2.0). Evaluated with JevBench by fstandhartinger (MIT) and the Decision Index kit by apolinario (MIT). Not affiliated with TypeSafe AI or Jev. See NOTICE.md and EVALUATION-PERMISSION.md.