Jev-Style-2B-Decision-v3-GGUF

Choose your starting point

Hosted API status · 2026-10-28: The hosted API is temporarily unavailable for maintenance. For a working first try, open the browser demo (2B v4), or run the open weights locally. Service information.

Try the latest 2B v4 in your browser →

Website: jevstyle.com · GitHub: jev-style · Collection: all v3 builds and demos

Jev-Style decision series: v1 · 2B → v2 · 2B → v3 · 0.8B → v3 · 2B

Quickest start: the jev-style package (0.3.0 or later) downloads these files and serves a local /v1/systemone API. It needs this repository's scorer, jev-score-v2, built once from build_jev_score.sh (see the Quick start below):

pip install jev-style
jev-style serve --release 2b-v3 --backend gguf --quant Q4_K_M --scorer /absolute/path/to/jev-score-v2

from jev_style import JevStyle, choice, noul

js = JevStyle.from_pretrained("chaoliangUNSW/Jev-Style-2B-Decision-v3-GGUF", quant="Q4_K_M",
                              scorer="/absolute/path/to/jev-score-v2")
out = js.decide("I was charged twice. Please fix this ASAP.", {
    "billing": noul("This ticket is about billing."),
    "tone": choice("What is the customer's tone?", ["calm", "frustrated", "angry"]),
})
print(out["answers"]["billing"]["noul"], out["answers"]["tone"]["choice"])

Jev-style decisions, now at 2B, for llama.cpp. These are the GGUF builds of Jev-Style-2B-Decision-v3: F16, Q8_0 and a 1.27 GB Q4_K_M, with a small bundled scorer (jev-score-v2) that reads the decision at one verdict slot per option. Full results, protocols, training data and licences are on the main model card.

_{These benchmark numbers were measured with the GGUF F16 file of this repository (pre-declared engine, one global temperature, each benchmark run once). JevBench: self-run with the official harness, not an official board entry; 95% CI 67.6–78.9% (Wilson). 73.6% is the highest JevBench public accuracy among the Qwen3.5-2B-family systems on the v1.4.1 board (decider-2b 71.0%, open-jev-zefan-2b 64.5%); decider-2b lies inside the CI. Jev is well ahead on JevBench and ahead on fin_topic. ¹ Jev numbers from the elcronos study (raw API), not re-run by us; on tweet_topic the 2B's macro-F1 (67.8%) is below Jev's (69.4%). Details: main card.}

25,600 tokens, no option cap. State, questions and every option share one 25,600-token budget; question and options over 2,048 tokens use a numbered-option catalogue. Nothing is truncated.

Files

• For long documents use Q8_0 or F16. Q4_K_M is noticeably noisier on long inputs.
• F16 is the runtime's default and the engine used for every benchmark number above.

_{Reference: HF FP32 on CPU, exact block attention, on the released bf16 checkpoint; gates declared before any format was scored. 1,000 real development rows (≤4,096 tokens) test agreement between formats. Long fixture: 35 requests / 43 questions up to 25,600 tokens, including catalogue-overflow questions and up to 151 options. ¹ For 4-bit the pre-declared gate is the accuracy drop (≤1.0 point); top-1 agreement is reported only. Sizes are the files (GB = 10^9 bytes). Converted and quantised with unmodified llama.cpp tools at commit 441df11f.}

Speed

Read once, then ask. On an Apple M1 Max (GGUF F16), the first question about a 24,501-token input took 16.2 s; a further question about the same state took 0.17 s, because the state is computed once and reused (medians). Ten questions about that state in one call took 16.8 s.

• Pick a GGUF file by size and accuracy (Files table above), not by these timings: the three files were timed one after another while the load from other jobs varied, so the gaps between the columns are within noise.

_{Apple M1 Max, 64 GB, macOS 15.7.5. Wall time around one decide / score_many call (tokenisation included), median of 3 calls with the state recomputed each time; the model was loaded beforehand (loading took 1.0–2.6 s here, not included). States: English documentation and source code of 878, 3,950, 24,436 tokens plus the question; 10 questions = 4 choice, 4 true/false and 2 score questions about the same state in one call; with the question and options each input was up to 943, 4,015 and 24,501 tokens. GGUF: jev-score-v2 on llama.cpp 441df11f, Metal, all layers on the GPU. Results were identical with and without a precomputed state. Other jobs shared the machine during these runs (1-minute load average 5.6–13.8 at the end of each run), so treat the numbers as indicative. All rows here were measured in one session (2026-09-27 03:11–03:21 AEST); an earlier run of the same rows (00:24–00:37 AEST, load average 8.2–16.6) was discarded because other jobs had slowed it (its times were up to 2.5× longer).}

Quick start

The files are standard Qwen3.5 text models, so llama.cpp loads them, but chat or text generation does not give you the model's decisions: this model needs block attention and a readout at one verdict slot per option. The bundled jev_score_v2.cpp (a small program on stock libllama) does exactly that. The runtime and jev-score binary of the 0.8B v3 repositories are not valid for this model.

pip install -U huggingface_hub
hf download chaoliangUNSW/Jev-Style-2B-Decision-v3-GGUF --local-dir jev-v3-2b-gguf
cd jev-v3-2b-gguf
pip install -r requirements.txt                 # tokenizers 0.23.2, numpy 2.5.3

# Build the scorer once against a llama.cpp checkout (tested at commit 441df11f65ea0b6d0c72965aaf70c8241070ddcb).
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp checkout 441df11f65ea0b6d0c72965aaf70c8241070ddcb
LLAMA_CPP=llama.cpp
sh build_jev_score.sh "$LLAMA_CPP"              # -> build/jev-score-v2 (or pass --jev-score PATH / set JEV_SCORE_V2_BIN)

One question (--quant picks the file: F16 default, Q8_0, Q4_K_M):

python jev_style_decision_gguf.py \
  --state "The user said: 'Write a reply to Bob with the Q3 report attached, but let me check it before it goes out.' The draft is written and the report is attached." \
  --question "What should the assistant do next?" \
  --options '{"send": "press Send now", "ask_user": "ask the user to review the draft", "discard": "delete the draft"}'
# -> "answer": "ask_user", probability 0.860 (send 0.128, discard 0.012; F16, Metal)

A state can be JSON (--state-json); with no --options the question is true/false; --qtype score takes a list of 2–10 levels, lowest first:

python jev_style_decision_gguf.py --quant Q8_0 \
  --state-json '{"app": "Calendar", "event": "Team sync", "start": "2026-10-02 15:00", "invitees": 14, "accepted": 3}' \
  --question "Is attendance for this meeting likely to be low?"
python jev_style_decision_gguf.py --quant Q4_K_M \
  --state "Ticket: the checkout page returns HTTP 500 for every customer since the last deploy." \
  --question "How urgent is this ticket?" --qtype score --options '["not urgent", "normal", "urgent", "critical"]'

The output has answer, probabilities (softmax of the scores at the calibrated temperature 0.828; --temperature 1.0 gives uncalibrated probabilities), the raw scores (logit(" yes") − logit(" no") per option), token counts and catalogue_overflow.

Many questions, as JSON lines (consecutive lines with the same state compute that state once; a malformed or over-budget line gets an "error" field, nothing is truncated):

cat > requests.jsonl <<'EOF'
{"id": "q1", "state": "Order #4411: payment received. Items are still in the warehouse waiting to be packed.", "question": "Has the order shipped?"}
{"id": "q2", "state": "Order #4411: payment received. Items are still in the warehouse waiting to be packed.", "question": {"t": "choice", "ins": "What is the order status?", "crit": {"awaiting_payment": null, "processing": "paid, not yet handed to the courier", "shipped": "handed to the courier"}}}
{"id": "q3", "state": "Hi, I was charged twice for my subscription this month.", "question": "Which intent is this?", "options": ["billing_question", "cancel_subscription", "technical_issue"]}
EOF
python jev_style_decision_gguf.py --quant Q8_0 --jsonl requests.jsonl

From Python:

from jev_style_decision_gguf import JevStyleDecisionGGUF

state = {"app": "Mail", "draft_to": "bob@example.com", "attachment": "Q3-report.pdf", "sent": False}
with JevStyleDecisionGGUF(quant="Q8_0") as model:
    one = model.decide(state, "Is a report attached to the draft?")
    many = model.score_many(state, [
        {"t": "noul", "ins": "Has the email been sent?", "crit": None},
        {"t": "choice", "ins": "What is attached?", "crit": {"report": "a report", "photo": "a photo", "nothing": None}},
    ])
print(one["answer"], round(one["top_probability"], 3))   # true 0.98 (Q8_0, Metal)
print([r["answer"] for r in many])                       # ['false', 'report']

• score_many(state, questions) computes the state once and reuses it for every question; the results are identical to calling decide per question.
• --ngl 0 runs on the CPU, with flash attention off by default (in a small CPU check, 6 short rows with Q8_0, it was slightly closer to the FP32 reference with flash attention off than on). --verify checks the sha256 of the tokenizer, readout_config.json and the selected GGUF against manifest.json before loading.
• The runtime checks the GGUF metadata and refuses GGUFs of a different architecture or size (e.g. the 0.8B v3); it does not tell this model apart from other Qwen3.5-2B GGUFs, so use the files from this repository.
• The input format, block attention and the readout are described on the main card.

Scope and limits

• Runtime required. Stock llama.cpp, Ollama and LM Studio can load these files but cannot produce the decision scores, and they would run causal attention. Use jev_style_decision_gguf.py with jev-score-v2.
• 25,600 tokens is the limit for the whole input (state + question + options + readout).
• Reduced-data training. The model was trained on a reduced data pool (60M tokens).
• Decision Index. Not run by us. The training pool includes the train splits of 7 of its benchmarks and format-imitating data for 7 more, so results on these are not zero-shot; the 14 benchmarks that are not zero-shot for this model are named on the main card.

Results charts

_{Protocol notes are under each chart and on the main card. Plotted values and sources: jevbench.data.json, zeroshot.data.json.}

Disclaimers and licence

Apache-2.0. Built on Qwen/Qwen3.5-2B (Apache-2.0); NOTICE lists the modifications. Some training data has restrictive or unclear terms (for example research-only jailbreak prompts and share-alike CC BY-SA sources), and some training rows are outputs of OpenAI GPT and Anthropic Claude models, whose providers' terms of use may restrict how models trained on them may be used. See Training data and licences on the main card. jev_score_v2.cpp is compiled against libllama (MIT); llama.cpp is not redistributed here. Not affiliated with, endorsed by or connected to TypeSafe AI or Jev (no Jev weights, code or outputs are used), the Laya authors or the Qwen team.

AI disclosure: code written with AI coding assistants (Claude Code) under my direction; I designed the project, trained the models and verified the results.

Contact

I welcome internship, employment, and research collaboration opportunities. Please contact me at yanchaoliang369@gmail.com.

欢迎提供实习、工作及科研合作机会，请邮件联系：yanchaoliang369@gmail.com。