Bonsai-JEV

A calibrated, image-capable 27B decision model that fits on an 8 GB GPU. The JEV recipe (a LoRA plus a 24-slot decision head, distilled to soft labels) trained on Ternary Bonsai 2 27B, Prism ML's ternary compression of Qwen3.8-27B. Ask a typed question (yes/no, 0–5 score, or 2–16-way choice) about a state that can include images; one forward pass returns calibrated probabilities over the options. No text is generated.

Code, training recipe and evaluation scripts: github.com/DakePeng/Bonsai-JEV · Mirror: ModelScope

The same weights also serve System 2: with the adapter switched off, the server is Ternary Bonsai 2 itself, byte for byte.

Status (2026-10-15): trained; all numbers below are measured through the deployment path (the fork running adapter.gguf on the ternary file). Decision Index: public vision suite 67.56 (level with d3-mini); text suite not run.

Independent project. Bonsai-JEV is built by an independent developer. It is not affiliated with, endorsed by, or sponsored by AutoTrust (autotrust), TypeSafe, or Prism ML. Their names appear only to credit their work and to identify the models and data this project builds on; all trademarks belong to their owners.

Built on

• Ternary Bonsai 2 27B by Prism ML (Apache 2.0). Created using Bonsai by Prism ML. Its ternary weights are the frozen base, unchanged and included here byte-identical; its llama.cpp fork is the runtime. Bonsai 2 is built from Qwen3.8-27B by Alibaba Cloud (Apache 2.0).
• JEV-9B and JEV-27B by autotrust (Apache 2.0). Training recipe, prompt template, verbalizers, the decision-head-as-lm_head-LoRA serving approach and the /v1/decide API are theirs; their published numbers are our targets.

At a glance

Results

Text decisions, test_set_30k of jev-distill-corpus-v3, KL(target ‖ model) after per-kind temperature, lower is better. "Jev-labelled" = the 25,376 rows labelled by TypeSafe Jev 1.13.

The other JEV-27B card metrics, same rows and definitions:

Fitted temperatures 1.014 / 1.014 / 1.012 (noul / score / choice): the model is calibrated by training. Validation KL fell from 0.070 at step 250 to 0.018 at step 4,000, flat over the last 250 steps.

Image decisions, VL-RewardBench (1,247 pairs, both orders averaged, the JEV demo protocol):

A tie with JEV-27B-VL (±2.4 at 1,247 pairs). One forward pass without averaging: 77.5 / 79.1 (as given / swapped).

Order-agree is the share of pairs whose verdict survives swapping the two answers; it shows whether a single forward pass can be trusted.

Embodied reasoning, ERQA (400 robot-scene questions, one pass, JEV-35B protocol): 52.0 (JEV-27B-VL 53.8, JEV-35B 51.3; chance ≈ 26, ±5 at 400 questions); 56.1 on single-image and 41.6 on multi-image questions.

Decision Index 0.3, public vision suite (12,210 rows, official kit, chance-corrected; compare with the board's public column): 67.56, level with vLLM-SR d3-mini 4B (67.67, ~9 GB) and 93% of JEV-27B-VL (72.80, ~56 GB); d3 27B 74.08, d3-flash 9B 69.87. Within 5% of JEV-27B-VL on documents, receipts, web pages and spatial questions (InfographicVQA 95.6 vs 96.7, KIE 90.0 vs 92.0, Mind2Web 75.3 vs 78.6, CV-Bench 73.3 vs 76.8); ~70% on multi-step visual reasoning (MMMU-Pro 28.8 vs 40.1, R-Bench-M 21.3 vs 31.1). Text suite not run. Single-request latency on one RTX PRO 6000 (the board's rule: median, mean and p80 ≤ 1 s): vision median 986 ms / p80 1,289 ms, text median 654 ms / mean 3,143 ms, so the model is not submitted to the board; the ternary fork is built for small memory, not large-GPU speed.

Agent judges (zero-shot, JEV-27B-DEMO prompts): AgentRewardBench AUROC 0.930 (JEV-27B-VL 0.91), precision / recall at 0.5 74.7 / 74.7 (78.4 / 70.2); Plan-RewardBench macro accuracy 74.7 (73.2). JevBench-231: 194 / 231 with JEV's bare-v1 prompt, +28 over the zero-shot base with the same prompt.

Speed and memory, RTX 4070 12 GB, PrismML llama.cpp fork:

PTQ1_0 runs the same adapter with identical decisions (100% top-1 agreement, zero probability difference on 3,000 rows) and is faster. Concurrency: 40 concurrent decisions over 8 slots match sequential ones (max |Δp| 0.0035, no flips; mixed chat unaffected); the fork batches only requests with the same LoRA set.

Other devices. Mac mini M4 16 GB (MLX): Prism's MLX 2-bit pack with this repo's peft/ and head.safetensors, no conversion (mlx_serve.py on GitHub); identical to the CUDA path on text (KL 0.00000 on 300 rows), 96.3% same verdict on images; 8.6–9.7 GiB; text p50 2.1 s, image p50 13.8 s. Radeon 780M iGPU (Vulkan): PTQ1_0 + fp16 adapter, ~7.7 GB of iGPU memory, prompt processing 54 tok/s; text p50 4.3 s, image p50 13.3 s, follow-up on a warmed scene 2.9 s.

Quickstart

Build the PrismML llama.cpp fork from source with CUDA (or Metal). Stock llama.cpp cannot load these files; on AVX-512 CPUs (e.g. AMD Zen 4) use a source build, the release binaries crash on load.

pip install fastapi uvicorn requests pydantic pillow
# 12 GB+ GPU: PQ2_0 base (~8.3 GB). Either source; ModelScope is faster from mainland China.
hf download dakebakery/Bonsai-JEV --exclude "*PTQ1_0*" --exclude "adapter/adapter_f16.gguf" --exclude "peft/*" --exclude "head.*" --local-dir Bonsai-JEV-weights
modelscope download --model dakepeng/Bonsai-JEV --exclude "*PTQ1_0*" "adapter/adapter_f16.gguf" "peft/*" "head.*" --local_dir Bonsai-JEV-weights
# start llama-server (:8080) + the /v1/decide wrapper (:8000); LLAMA_BIN = the fork's build/bin
LLAMA_BIN=~/llama.cpp/build/bin bash Bonsai-JEV-weights/run_decide.sh start Bonsai-JEV-weights

# 8 GB GPU: PTQ1_0 base + fp16 adapter (~6.8 GB download)
hf download dakebakery/Bonsai-JEV --exclude "*PQ2_0*" --exclude "adapter/adapter.gguf" --exclude "peft/*" --exclude "head.*" --local-dir Bonsai-JEV-weights
BASE_GGUF=Ternary-Bonsai-2-27B-PTQ1_0.gguf ADAPTER_FILE=adapter_f16.gguf CTX=4096 NP=1 \
  LLAMA_BIN=~/llama.cpp/build/bin bash Bonsai-JEV-weights/run_decide.sh start Bonsai-JEV-weights

System 1: typed decisions

curl -s localhost:8000/v1/decide -H 'content-type: application/json' -d '{
  "kind": "choice",
  "state": ["Camera frame: ", {"image": "frame.jpg"}, "\nThe gripper is open above the table."],
  "question": "Which object should the arm pick first?",
  "options": ["red cube", "blue cube", "nothing, the scene is unsafe"]}'
# {"kind":"choice","options":[...],"probabilities":[0.71,0.22,0.07],"choice_index":0,"choice":"red cube",...}

• kind: noul (options false/true), score (0–5), choice (2–16 options, labelled A–P).
• state: a string, a JSON object, or a list mixing text and {"image": path | URL | data-URL}.
• The response shape matches JEV's serve_decide.py, so JEV's jev_client.py works with JEV_URL=http://localhost:8000 JEV_BACKEND=decide.
• Several questions about one scene: POST /v1/warm {"state": ..., "kinds": ["noul"]} once, then each /v1/decide evaluates only its question tokens (~0.2 s on the 4070). The model has recurrent layers, so prefix reuse needs this explicit boundary, one per kind.

System 2: chat and reasoning

Send ordinary OpenAI-style requests to llama-server on port 8080. run_decide.sh loads the adapter with global scale 0, so these requests see the untouched Ternary Bonsai 2: with the adapter off, 12/12 test answers are byte-identical to a server without the adapter. Do not set the adapter's scale to 1 for chat; with it on, answers change.

Prompt format

The wrapper renders JEV's bare-v1 template; you only pass the fields:

[kind] choice
[state] <state, images inline>
[question] <question>
[options]
A) <option 1>
B) <option 2>
[decision]:

The probabilities are read from the verbalizer tokens at [decision]: (false/true, 0–5, A–P), plus the head bias and per-kind temperature from decision.json. Put the scene first and the question last so warmed prefixes are reused.

Files

The adapter is bound to this exact base file (Hadamard rotation metadata, value-head layout). Use the bundled GGUF rather than a newer or older Bonsai 2 file.

Training

LoRA r=16, α=32, dropout 0.05 on in_proj_qkv, in_proj_z, out_proj, q/k/v/o_proj, gate/up/down_proj of all 64 layers, applied to the raw (un-rotated) layer input exactly as the fork applies it; fp32 decision head 5120→24 initialised from the lm_head rows of the verbalizer tokens. Loss KL(target ‖ model) over active slots + 0.5·RPS for scores; 30% of choice rows shuffled. 128 rows/step stratified by kind, micro-batches ≤ 24k padded tokens, gradient checkpointing, AdamW (0.9, 0.98), LR 2e-4 head / 1e-4 LoRA, 3% warmup, cosine to 0.02×, 4,000 steps (JEV-27B's 5,000 steps minus its 21% placeholder rows). 518,603 training rows of jev-distill-corpus-v3 (the 137,203 yuri_v1 rows with placeholder [0.5, 0.5] labels dropped rather than weighted ×0.05 as JEV-27B does). Dropping them costs nothing measurable: noul KL 0.0041 vs JEV-27B's 0.004, fitted noul temperature 1.014 on both. Base frozen in BF16, dequantised from the ternary file. 2× RTX PRO 6000 96 GB, ~24 h. Release checkpoint: validation KL tied at steps 3,750 and 4,000 (0.0180), so the tie was broken on the calibration split, where step 4,000 is lower on all three kinds; the test set was not used to choose; per-kind temperatures fitted by L-BFGS on the calibration split.

Code, alignment checks and the full log of decisions: GitHub.

Limitations

• Quality ceiling is JEV level. The ternary base can only take a LoRA. Full fine-tunes of Qwen3.8-27B score higher on the Jev Decision Index; this model is about memory, not about beating them.
• Same teacher as JEV, so on text it matches JEV-27B rather than exceeding it. Transfer to unseen task families is weaker (KL 0.128 vs 0.104).
• Ternarisation costs more on vision than on text (Bonsai's card: MMMU-Pro 81.7 → 75.5). Image decisions are zero-shot, as in JEV-27B-VL.
• English prompts. The training data is English; decisions on other languages are untested.
• Latency: each decision is a 27B forward pass; on an RTX PRO 6000 it is 2–4× slower than JEV-27B-VL on vLLM (above the Decision Index's 1 s rule); on Jetson Orin NX expect seconds per image decision.
• Visual reasoning is the weakest area (~70% of JEV-27B-VL on MMMU-Pro and R-Bench-M).

License and acknowledgements

Apache 2.0 for the adapter, head and scripts. Created using Bonsai by Prism ML; Ternary Bonsai 2 27B is © Prism ML, Inc., Apache 2.0, built from Qwen3.8-27B (© Alibaba Cloud, Apache 2.0) and redistributed unchanged with its license and NOTICE. Recipe and serving design from autotrust's JEV-9B / JEV-27B (Apache 2.0). Trained on SargeDev/jev-distill-corpus-v3 (Apache 2.0; openjev_v2 CC0), the same corpus JEV-9B and JEV-27B were trained on; its yuri_v3 labels are outputs of TypeSafe's Jev 1.13, so commercial users may want to review TypeSafe's API terms. VL-RewardBench is used for evaluation only.

Thanks to Prism ML, autotrust, SargeDev, the Qwen team and the maintainers of the Jev Decision Index.

Also on ModelScope

modelscope.cn/models/dakepeng/Bonsai-JEV, same files, faster from mainland China.