Decision 4B v1.4 (FlyMyJev-4B, preview)
A new training run (qwen35_4b_letter_h12_v66c) on a larger mixture than v1.2, with the per-request-type readout of
v1.3: yes/no (noul) requests use their own, lower temperature so that an answer commits when the model leans one way.
An open model for typed decisions: give it a state (a ticket, a policy and a case, a log, an answer to judge) and a
question with a closed set of answers (noul yes/no, choice, score), and it returns a probability for every
declared option in one forward pass. Nothing is generated. Independent project: not a release of TypeSafe, not
affiliated with it, not a reconstruction of Jev's closed implementation.
Run qwen35_4b_letter_h12_v66c. Every number below is our own measurement; we make no claim about a JevBench rank.
What it is
| Base | Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (Apache-2.0), downloaded separately |
| Adapter | LoRA rank 16, alpha 32, dropout 0.05 on the attention projections of both layer kinds (q_proj, k_proj, v_proj, o_proj, in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj); 14.4M trainable parameters |
| Prompt | the chat template with thinking disabled; one JSON user message {evidence, criterion, options[{letter, description}]}, descriptions "<key>: <text>", yes/no options true then false (SemIf's format, MIT) |
| Readout | the option letters' logits at the last prompt token (fp32 projection of its hidden state), softmax at the temperature in model.json for the request type: choice and score 1.40 (NLL on our own data only), yes/no 0.40 (chosen on our own held-out yes/no items so that answers stay decisive) |
| Output | a distribution over exactly the declared options: a label outside that set cannot come out |
| Serving | the LoRA is folded into the base at load; one CUDA graph per padded input length; p50 18.3 ms / p95 21.5 ms per short decision on one RTX 4090 (eager 24 ms); the same answer as the eager path on 120/120 checked items |
| Inputs | up to 16,384 tokens |
Requirements
- A C compiler at run time (for example
build-essential): the flash-linear-attention Triton kernels are compiled on the first request, and CUDA-graph capture at load needs them too. model.load()verifies every file againstmanifest.jsonand refuses symlinks, so a Hugging Face cache snapshot (whose files are links intoblobs/) must be copied to a real directory first, for example withhuggingface-cli download --local-dir.
Use
import model # this repository's model.py
m = model.load() # verifies every file against manifest.json, downloads the pinned base on first use
m.decide("Refunds need a receipt and a purchase within 30 days. The customer bought 12 days ago and has no receipt.",
{"type": "noul", "instructions": "Is a refund permitted under the policy?"})
# {'probabilities': {'true': ..., 'false': ...}, 'input_tokens': ..., 'seconds': ...}
JevBench: server.py serves TypeSafe's /v1/systemone wire format for the typesafe adapter; jevbench_adapter.py
is an in-process adapter; run_jevbench.py runs the official runner without editing its registry. Set
FLYMYJEV_DEVICE=cpu to run on CPU (slow, no CUDA graphs).
Results (our measurements)
The frozen base and this adapter were measured on the same GPU in the same session with the same prompt; changes are paired (fixed / broken items). JevBench's judge tier and sealed set are not public and are not included.
| set | frozen Qwen3.5-4B | this model | fixed / broken | reference |
|---|---|---|---|---|
| JevBench public, hard (111) | 60.4 | 75.7 | 24 / 7 | Jev 1.13: 74.1 on the full hard tier (220 items, 109 held out) |
| JevBench public, standard (72) | 97.2 | 100.0 | 2 / 0 | |
| JevBench public, easy (48) | 97.9 | 100.0 | 1 / 0 | |
| JevBench public, all 231 | 79.7 | 88.3 | 27 / 7 | Jev 1.13: 86.6 |
| our hard dev set (160, 8 families) | 42.5 | 65.0 | 51 / 15 | Jev 1.13: 57.5 |
| our real-case sets v1-v3 (407) | 88.9 | 89.7 | 17 / 14 | Jev 1.13: 94.6 / 97.8 / 98.9 on v1 / v2 / v3 |
- Hard dev set by family (frozen -> this model): date and number decisions 10 -> 60, adversarial traps 50 -> 95, abstention 60 -> 95, paraphrase robustness 55 -> 85, multi-hop lookups 30 -> 50, trade-offs 25 -> 45, answer judging 55 -> 45, long policies 55 -> 45 (20 items per family).
- Calibration on the public hard tier at the served temperature: ECE 0.071, fidelity to exact gold distributions 86.3 (1 - mean total variation distance).
- Real-case set v3 puts the gold at option A in 46 % of its choice items; the frozen base's first-option habit scores there, and training removes it. Read v3 with that in mind.
Training
One run from the base, no merging of runs and no ensembles: 2 epochs, 996 steps of at most 12,288 padded tokens (11.1M tokens), lr 3e-05 with warm-up and cosine decay, cross-entropy on the option-letter logits (the exact distribution where an item declares one), choice options shuffled on every pass, seed 99. The recipe shape follows JevK5 v0.2 (allebee/jevk5, Apache-2.0).
Data: our own decision data and licence-reviewed public datasets. No JevBench item and no output of Jev was used for training, tuning or model selection.
Benchmark hygiene
- The temperature was fitted on our own calibration split and hard dev set, never on benchmark items.
- The gates that made this run a release candidate were declared before the run.
- The JevBench public items were used to measure, and once to diagnose: comparing our frozen outcomes with SemIf's published per-item outcomes showed that our first yes/no rendering lost about 5 points on yes/no items; we adopted SemIf's rendering everywhere. No training data was chosen or built from public items.
Limitations
- One pass of a 4B model: no step-by-step reasoning. Chained arithmetic and long multi-hop documents stay weak.
- Messages where two options are both true under a one-answer schema are untested.
- English only; the calibration was fitted on our own data.
Licence and credits
Adapter, code and configuration: Apache-2.0 (LICENSE). Base: Qwen3.5-4B by the Qwen team (Apache-2.0). Prompt format: SemIf by TheoLeeCJ (MIT). CUDA-graph runtime design and recipe shape: JevK5 by allebee (Apache-2.0). Evaluated with JevBench by fstandhartinger (MIT). Not affiliated with TypeSafe AI or Jev. See NOTICE.md and EVALUATION-PERMISSION.md.