Decision 4B v1.1 (FlyMyJev-4B, preview)
An open model for typed decisions: give it a state (a ticket, a policy and a case, a log, an answer to judge) and a
question with a closed set of answers (noul yes/no, choice, score), and it returns a probability for every
declared option in one forward pass. Nothing is generated. Independent project: not a release of TypeSafe, not
affiliated with it, not a reconstruction of Jev's closed implementation.
Run qwen35_4b_letter_h10_v63. Every number below is our own measurement; we make no claim about a JevBench rank.
What it is
| Base | Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (Apache-2.0), downloaded separately |
| Adapter | LoRA rank 16, alpha 32, dropout 0.05 on the attention projections of both layer kinds (q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj); 14.4M trainable parameters |
| Prompt | the chat template with thinking disabled; one JSON user message {evidence, criterion, options[{letter, description}]}, descriptions "<key>: <text>", yes/no options true then false (SemIf's format, MIT) |
| Readout | the option letters' logits at the last prompt token (fp32 projection of its hidden state), softmax at the temperature in model.json (1.35, fitted on our own data only) |
| Output | a distribution over exactly the declared options: a label outside that set cannot come out |
| Serving | the LoRA is folded into the base at load; one CUDA graph per padded input length; p50 18.6 ms / p95 21.8 ms per short decision on one RTX 4090 (eager 59 ms); the same answer as the eager path on 120/120 checked items |
| Inputs | up to 16,384 tokens |
Use
import model # this repository's model.py
m = model.load() # verifies every file against manifest.json, downloads the pinned base on first use
m.decide("Refunds need a receipt and a purchase within 30 days. The customer bought 12 days ago and has no receipt.",
{"type": "noul", "instructions": "Is a refund permitted under the policy?"})
# {'probabilities': {'true': ..., 'false': ...}, 'input_tokens': ..., 'seconds': ...}
JevBench: server.py serves TypeSafe's /v1/systemone wire format for the typesafe adapter; jevbench_adapter.py
is an in-process adapter; run_jevbench.py runs the official runner without editing its registry. Set
FLYMYJEV_DEVICE=cpu to run on CPU (slow, no CUDA graphs).
Results (our measurements)
The frozen base and this adapter were measured on the same GPU in the same session with the same prompt; changes are paired (fixed / broken items). JevBench's judge tier and sealed set are not public and are not included.
| set | frozen Qwen3.5-4B | this model | fixed / broken | reference |
|---|---|---|---|---|
| JevBench public, hard (111) | 60.4 | 75.7 | 23 / 6 | Jev 1.13: 74.1 on the full hard tier (220 items, 109 held out) |
| JevBench public, standard (72) | 97.2 | 97.2 | 2 / 2 | |
| JevBench public, easy (48) | 97.9 | 100.0 | 1 / 0 | |
| JevBench public, all 231 | 79.7 | 87.4 | 26 / 8 | Jev 1.13: 86.6 |
| our hard dev set (160, 8 families) | 42.5 | 60.0 | 42 / 14 | Jev 1.13: 57.5 |
| our real-case sets v1-v3 (407) | 88.5 | 88.7 | 14 / 13 | Jev 1.13: 94.6 / 97.8 / 98.9 on v1 / v2 / v3 |
- Hard dev set by family (frozen -> this model): date and number decisions 10 -> 60, abstention 60 -> 90, adversarial traps 50 -> 80, paraphrase robustness 55 -> 75, trade-offs 25 -> 45, multi-hop lookups 30 -> 45, answer judging 55 -> 45, long policies 55 -> 40 (20 items per family).
- Calibration on the public hard tier at the served temperature: ECE 0.065, fidelity to exact gold distributions 87.0 (1 - mean total variation distance).
- Real-case set v3 puts the gold at option A in 46 % of its choice items; the frozen base's first-option habit scores there, and training removes it. Read v3 with that in mind.
Package check (25 Sep 2026)
This package, as published, was run on one RTX 4090 against the JevBench v1.4.2 harness on the 231 public items: the official runner with jevbench_adapter.py and server.py with the harness's typesafe adapter both read 203/231 (easy 48/48, standard 70/72, hard 85/111). Details: package_check.json.
Training
One run from the base, no merging of runs and no ensembles: 2 epochs, 674 steps of at most 12,288 padded tokens (7.3M tokens), lr 3e-05 with warm-up and cosine decay, cross-entropy on the option-letter logits (the exact distribution where an item declares one), choice options shuffled on every pass, seed 99. The recipe shape follows JevK5 v0.2 (allebee/jevk5, Apache-2.0).
Data: our own decision data and licence-reviewed public datasets. No JevBench item and no output of Jev was used for training, tuning or model selection.
Benchmark hygiene
- The temperature was fitted on our own calibration split and hard dev set, never on benchmark items.
- The gates that made this run a release candidate were declared before the run.
- The JevBench public items were used to measure, and once to diagnose: comparing our frozen outcomes with SemIf's published per-item outcomes showed that our first yes/no rendering lost about 5 points on yes/no items; we adopted SemIf's rendering everywhere. No training data was chosen or built from public items.
Limitations
- One pass of a 4B model: no step-by-step reasoning. Chained arithmetic and long multi-hop documents stay weak.
- Messages where two options are both true under a one-answer schema are untested.
- English only; the calibration was fitted on our own data.
Licence and credits
Adapter, code and configuration: Apache-2.0 (LICENSE). Base: Qwen3.5-4B by the Qwen team (Apache-2.0). Prompt format: SemIf by TheoLeeCJ (MIT). CUDA-graph runtime design and recipe shape: JevK5 by allebee (Apache-2.0). Evaluated with JevBench by fstandhartinger (MIT). Not affiliated with TypeSafe AI or Jev. See NOTICE.md and EVALUATION-PERMISSION.md.