autotrust/JEV-27B-VL running on a single Intel Arc Pro B70 (32 GB). The original card asks for a GPU with 80 GB or more.
Made by @imdariotoo. I post my B70 and local-model releases on X, so follow there for the next ones.
JEV-27B-VL is the unmodified Qwen3.8-27B (its weights are byte-identical to Qwen/Qwen3.8-27B) plus a rank-32 LoRA,
a decision head and a calibration file. So this repo is:
adapter_vllm/ (LoRA + decision head), calibration.json, serve_decide.py
(apache-2.0, by autotrust)The LoRA is applied at runtime, not merged. That is how JEV is meant to run: System 2 is the plain base model, and System 1 is the same engine with the adapter switched on, so both come from one server.
autotrust/JEV-27B is the same model without the vision tower. I checked: all 851 of its backbone tensors are byte-identical to Qwen3.8-27B, and its adapter's 802 tensors are byte-identical to the one in this repo. Its decision head and calibration file are identical too. So this repo is the B70 port of both. PubMedQA below is in fact compared against JEV-27B's own BF16 run.
One trap if you mix files: JEV-27B's adapter names its layers model.layers.*, while this backbone loads as the
multimodal class (model.language_model.layers.*). vLLM then applies only the adapter's lm_head part, without any
error, and System 1 quietly gets worse (PubMedQA 74.4% with 16 confident answers flipped, instead of 78.4% and 0).
Use the adapter_vllm/ shipped here, which already has the right names.
I wrote the pass rules down before running anything. All checks are text-only.
| check | int4 on B70 | reference | rule | result |
|---|---|---|---|---|
| card example (billing / shipping / tech support) | 0.9954 / 0.0000 / 0.0046 | 0.9969 / 0.0000 / 0.0031 (card) | – | matches |
| PubMedQA, 500 questions, accuracy | 78.4% | 77.8% (JEV's BF16 run) | within 2 points | pass |
| PubMedQA, same answer as BF16 | 97.6% | – | – | – |
| PubMedQA, flipped answers where BF16 was ≥ 80% sure | 0 of 333 | – | ≤ 1% | pass |
| held-out yes/no (500 rows), AUROC on rows whose target isn't near 50/50 | 1.000 | 0.995 (card, full 30K, rows not stated) | ≥ 0.985 | pass |
| held-out ratings 0-5 (500 rows), MAE of expected rating | 0.107 | 0.098 (card) | ≤ 0.15 | pass |
| held-out choice, all 9,199 rows, top-1 agreement on clear rows | 94.6% (7,305 / 7,723) | 95.8% (card) | ≥ 93% | pass |
Notes, so nothing is hidden:
08-biomedical-qa/results.json). It was made
with the text model JEV-27B, so the 2.4% disagreement mixes int4 error with the VL-vs-text difference (the JEV card
puts that at a mean |Δp| of 0.010).thinking: "on" / "auto"; only its speed was measured),
contexts above 131K.Raw per-item outputs and the eval script are in eval/. The full-set choice run and the 1,500-row check used eager
mode; PubMedQA and the 1,500-row check were repeated in the fast mode with the same results.
thinking: "on" on 32 PubMedQA questions, 8 at a time: median 469 thinking tokens and 17.7 s per
question, 170 tok/s across the batch. Eager mode without MTP (FAST=0) does 8 tok/s.GPU_UTIL=0.90 with a 131K max context.GPU_UTIL=0.72 keeps vLLM at 21.8 GiB and still leaves 4.6 GiB of KV cache (≈ 138K tokens), so
the full 131K context fits. Only 16 of the 64 layers keep a KV cache, the rest are linear attention. Checked on the
B70 under that cap (the example decision gives the same probabilities to every digit); not run on a real 24 GB card.Needs Docker with /dev/dri, one Arc Pro B70 (or another 32 GB Battlemage card, untested), about 12 GB of free host
RAM, and my B70 cookbook for the vLLM patches and attention
kernels. Everything is pinned to the vllm/vllm-openai-xpu:v0.31.0 image. On a 24 GB card add GPU_UTIL=0.72. The launcher starts in the fast mode (XPU graphs + MTP); FAST=0 gives eager mode.
git clone -b b70.6 https://github.com/riceharvest/b70-qwen38-cookbook # tag b70.6 = the v0.31 patch scripts this launcher uses
(cd b70-qwen38-cookbook && gh release download v3 -p 'libattn_kernels_xe_2.b70.so' -D kernels) # v2+: fixes a packed-MTP-verify bug in v1
hf download darioooooo0o/JEV-27B-VL-GPTQ-Int4-asym --local-dir ~/models/jev-b70
COOKBOOK=$PWD/b70-qwen38-cookbook MODEL_DIR=~/models/jev-b70 bash ~/models/jev-b70/serve-b70.sh # :8000
curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{"kind":"choice",
"state":"Customer: my card was charged twice for one coffee.","question":"Which team should handle this?",
"options":["billing","shipping","tech support"]}'
The prebuilt image ghcr.io/riceharvest/b70-vllm:v0.31.0-b70.3 also serves it (use that tag; :latest moved to vLLM v0.32, not tested with JEV yet): it detects a JEV repo
(-e MODEL=darioooooo0o/JEV-27B-VL-GPTQ-Int4-asym) and starts System 1 + System 2 with the LoRA.
The API is JEV's own (/v1/decide, or model jev-decision through /v1/completions), so the
JEV card and its demo repo
apply as they are. Keep --max-num-seqs 8: JEV notes that with more than 8 sequences per batch, vLLM's LoRA path
returns wrong System 1 probabilities.
cd eval
git clone https://github.com/yuhai-china/JEV-27B-DEMO demo
mkdir -p data && hf download SargeDev/jev-distill-corpus-v3 test_set_30k.jsonl --repo-type dataset --local-dir data
JEV_URL=http://localhost:8000 python3 eval_jev.py pubmedqa
JEV_URL=http://localhost:8000 python3 eval_jev.py s1 500
JEV_URL=http://localhost:8000 python3 eval_jev.py choice_full
JEV-27B-VL, its adapter, decision head, calibration and serve_decide.py are by
autotrust (apache-2.0). The backbone is Qwen3.8-27B by the Qwen team (apache-2.0).
The MTP and prefix-cache patches in the cookbook are from
SergiioB's B70 cookbook (MIT).
Questions or issues: reply to @imdariotoo on X or open a discussion on this repo.
model-extra0.safetensors 5.1 GBmodel-extra1.safetensors 1.8 GBadapter_vllm/adapter_model.safetensors 452 MBmodel-layer10.safetensors 200 MBmodel-layer12.safetensors 200 MBmodel-layer13.safetensors 200 MBmodel-layer14.safetensors 200 MBmodel-layer16.safetensors 200 MBmodel-layer17.safetensors 200 MBmodel-layer18.safetensors 200 MBmodel-layer20.safetensors 200 MBmodel-layer21.safetensors 200 MBmodel-layer22.safetensors 200 MBmodel-layer24.safetensors 200 MBmodel-layer25.safetensors 200 MBmodel-layer26.safetensors 200 MBmodel-layer28.safetensors 200 MBmodel-layer29.safetensors 200 MBmodel-layer30.safetensors 200 MBmodel-layer32.safetensors 200 MBmodel-layer33.safetensors 200 MBmodel-layer34.safetensors 200 MBmodel-layer36.safetensors 200 MBmodel-layer37.safetensors 200 MBmodel-layer38.safetensors 200 MBmodel-layer40.safetensors 200 MBmodel-layer41.safetensors 200 MBmodel-layer42.safetensors 200 MBmodel-layer44.safetensors 200 MBmodel-layer45.safetensors 200 MBmodel-layer46.safetensors 200 MBmodel-layer48.safetensors 200 MBmodel-layer49.safetensors 200 MBmodel-layer50.safetensors 200 MBmodel-layer52.safetensors 200 MBmodel-layer53.safetensors 200 MBmodel-layer54.safetensors 200 MBmodel-layer56.safetensors 200 MBmodel-layer57.safetensors 200 MBmodel-layer58.safetensors 200 MBmodel-layer60.safetensors 200 MBmodel-layer61.safetensors 200 MBmodel-layer62.safetensors 200 MBmodel-layer00.safetensors 200 MBmodel-layer01.safetensors 200 MBmodel-layer02.safetensors 200 MBmodel-layer04.safetensors 200 MBmodel-layer05.safetensors 200 MBmodel-layer06.safetensors 200 MBmodel-layer08.safetensors 200 MBmodel-layer09.safetensors 200 MBmodel-layer11.safetensors 193 MBmodel-layer15.safetensors 193 MBmodel-layer19.safetensors 193 MBmodel-layer23.safetensors 193 MBmodel-layer27.safetensors 193 MBmodel-layer31.safetensors 193 MBmodel-layer35.safetensors 193 MBmodel-layer39.safetensors 193 MBmodel-layer43.safetensors 193 MBmodel-layer47.safetensors 193 MBmodel-layer51.safetensors 193 MBmodel-layer55.safetensors 193 MBmodel-layer59.safetensors 193 MBmodel-layer63.safetensors 193 MBmodel-layer03.safetensors 193 MBmodel-layer07.safetensors 193 MB有文件?提交种子。
提交种子autotrust/JEV-27B-VL