OpenBonsai Qwen3.5-9B (ternary, PTQ1_0)

Research preview. The model is useful for experiments, not a finished product. This is the second release, made with more code in the data. On the frozen 50-problem HumanEval subset it passes 34 problems with thinking off, against 19 for the first release. That subset was also used to pick it from four runs, and its training data restates some of those tasks in other words, so 34/50 probably overstates the gain on new problems. On 114 HumanEval problems outside that subset, which played no part in the choice, it passes 62 against the first release's 45; a screen flagged 34 of them as the same task as, or related to, one in its traces, and on the other 80 it passes 38 against 26 (see Results). Its math and code are still below the original model and Prism ML's Ternary Bonsai 2 9B, and it fits general text slightly worse than the first release.

Independent project, not affiliated with or endorsed by Prism ML. Bonsai is Prism ML's product.

This is Qwen/Qwen3.5-9B compressed to ternary weights by OpenBonsai, an open (Apache-2.0) compression framework.

• Weights: {-1, 0, +1}, in groups of 128, each group with one fp16 scale.
• File: OpenBonsai-Qwen3.5-9B-PTQ1_0.gguf, 1,997,486,976 bytes (about 2 GB), SHA-256 818d8783bc39d92b2777a0333d38f7a25021a7b906f47344b83b7109eba6f7a0.
• Version: the second release, made on 28 October 2026 with examples/qwen3_5-9b-reasoning-code.yaml. The first release (SHA-256 4f2f3d1069a505059126e13ed1d2d7928e715930fe83f6d284426c4b19e35532) is still available at revision 40964de, tag reasoning-mix-v1. Both files have the same name and the same general.name, so only the SHA-256 tells them apart.
• Layout: the same tensor layout as Prism ML's Ternary-Bonsai-2-9B. All 202 large matrices are ternary, including the embedding and output head.
• Text only: no vision tower and no MTP layers.

Running it

It runs in Prism's llama.cpp fork and in ik_llama.cpp. Upstream llama.cpp does not support PTQ1_0. Prism's fork was tested at commit c9c34bc and ik_llama.cpp at 4955387, both as CPU builds; the CUDA builds are untested.

llama-cli -m OpenBonsai-Qwen3.5-9B-PTQ1_0.gguf -p "Explain why the sky is blue." -n 256

The chat template and tokenizer are embedded in the GGUF. Qwen3.5's chat template turns thinking on unless the caller turns it off. With thinking on, this model often thinks past 4,096 tokens (see Results).

Prism's runtime quantizes activations to 8 bits, so its output can differ slightly from PyTorch's. A check on fixed text compared Prism's runtime with an FP32 PyTorch reference of the same weights: on WikiText-2 test (8 chunks of 512 tokens, 2,040 scored positions), the two put the same token first at 99.2% of positions, and both give perplexity 14.15 (research/runs/2026-10-29-runtime-parity-9b/). This model's GSM8K and HumanEval scores under Results come from the PyTorch harness in BF16 on the GPU, which this check does not cover, and they are generated answers, where one differing token can change the rest.

ik_llama.cpp added PTQ1_0 in PR 2550, by Bl4ckyou, who also reported that this model runs there (openbonsai#13): about 2 GB of RAM and about 10 tokens/s of generation on an i5-13420H laptop CPU, with 4 threads and -rtr 1. On the same text, ik_llama.cpp gives perplexity 14.15 with -rtr 1 and 14.14 without, and puts the same token first as Prism's runtime at 99.0% of the 2,040 positions; greedy replies to three chat prompts were identical in both runtimes. It applies the file's own Hadamard sign vectors, which differ from those of Prism's Ternary-Bonsai-2-9B; with the rotation switched off, perplexity is in the millions (research/runs/2026-11-02-ik-llama-cpp/). -rtr 1 (run-time repack) rearranges the ternary weights into 8-row blocks while the model loads, for ik_llama.cpp's fast CPU kernels:

llama-server -m OpenBonsai-Qwen3.5-9B-PTQ1_0.gguf -rtr 1 -t 4 --port 8080

How it was made

One RTX 5090, with openbonsai run --config examples/qwen3_5-9b-reasoning-code.yaml. A run from scratch takes about 5 hours, estimated from the measured stages; the measured run reused cached traces and stages from earlier runs with the same settings, so it took less GPU time.

1. Self-generated traces: the BF16 model answers 2,400 prompts (2.51M tokens, 2.10M of them generated). The prompts come from the training splits of GSM8K, MATH, MBPP and OpenCodeInstruct, plus UltraChat, with thinking on and off. Code is 50% of the generated tokens, against 32% in the first release (1,520 prompts). This is the AYOT idea from ScaleQ-1.58 (arXiv 2608.01078). A contamination guard refuses GSM8K and HumanEval test problems and leaves out training prompts that share a 13-word run with one. It does not catch a test task restated in other words (see Results).
2. Rotation: sign + Hadamard rotation in Prism's layout.
3. Ternary rounding: sequential GPTQ, on 256 × 2,048 tokens. 192 of the 256 blocks are traces, 32 are WikiText-2 train and 32 are UltraChat.
4. Ternary embedding and head: also via GPTQ.
5. Scale distillation: the per-group scales are distilled end to end against the BF16 model's top-64 outputs, with the codes frozen. This uses one epoch over 3.8M tokens (1,856 sequences of 2,048), and 59% of them are traces.
6. Export and checks: a verified PTQ1_0 GGUF export, which Prism's runtime loads.

This is not Prism ML's recipe, which has not been published. Only the file format is shared. Method notes are in the repository. The runs behind this version are in research/runs/2026-10-28-humaneval-gap/, and its check on fresh problems in research/runs/2026-10-29-confirming-eval/.

Results

Every GSM8K and HumanEval score, Prism's model's included, comes from OpenBonsai's PyTorch harness, not from Prism's runtime. Prism's model and the generic-calibration model were scored from their GGUF files; this model and the first release from the state each GGUF was exported from, whose codes and scales the export check matched bit for bit. All rows are one run each, on small evaluation sets.

• Reasoning: greedy decoding with the model's chat template.
  • "Off" means thinking off: a frozen 100-problem GSM8K subset and a frozen 50-problem HumanEval subset, with a 512-token cap.
  • "On" means thinking on, the template's default: the first 50 and 25 of those subsets, with a 4,096-token cap.
• Dev NLL: 2,048 WikiText-2 test tokens. ARC: 100 ARC-Easy questions.
• KL and EAR: against BF16 over 4,088 WikiText-2 test tokens. KL is the full-vocabulary KL(BF16 ‖ model). EAR is the expected acceptance rate, Σ min(p, q). The values for Prism's model and the generic-calibration model are in research/runs/2026-10-27-compare-ear-v1/.

Code

• More code in the data raised HumanEval with thinking off from 19/50 to 34/50 on the frozen subset. 18 problems were solved only by this model and 3 only by the first release (two-sided sign test p = 0.0015). With thinking on, HumanEval went from 10/25 to 14/25 (6 problems against 2, p = 0.29).
• Chosen from four runs. This model is one of four runs with different data and epoch counts. The 1-epoch setting was picked on the same 50 HumanEval and 100 GSM8K problems it is reported on, so 34/50 probably overstates its gain on new problems.
• Reworded test tasks. The 13-word guard only skips trace prompts that are near duplicates of a test problem; tasks restated in other words are not caught. A later screen found that this model's traces restate at least 6 of the 50 frozen HumanEval tasks (HumanEval/13, 20, 23, 35, 61 and 90), all six in its distillation data; the first release's distillation data already had 23 and 61. The screen can miss restatements that share few words. Without those six and two related tasks (111 and 156), this model passes 28/42 and the first release 15/42 (16 problems against 3, p = 0.0044).
• Code is still the clearest weakness: 34/50 with thinking off, against 44 for BF16 and 45 for Prism's model. Of the 44 problems BF16 solves, this model misses 12 (the first release 25).

Check on fresh problems

A comparison with the first release on problems outside the frozen subsets: HumanEval positions 51-164 and GSM8K positions 101-1,319 of the same seeded order with thinking off, and HumanEval positions 26-75 with thinking on (26-50 were scored only with thinking off during the selection). Its decision rule, and a screen of HumanEval positions 51-164 for tasks restated in the traces, were fixed before any score was read. The pre-registration (PREREGISTRATION.md, which states the rule), the per-problem results and the rule applied are in research/runs/2026-10-29-confirming-eval/; the same rule, written as four numbered conditions, was under Caveats in research/runs/2026-10-28-humaneval-gap/ before the first score.

With thinking off, on the 114 fresh HumanEval problems this model passes 62 and the first release 45: 24 problems were solved only by this model and 7 only by the first release (one-sided sign test p = 0.0017), a difference of 14.9 percentage points (95% CI 5.7 to 24.1). That is about half the 30 points on the frozen subset. These problems played no part in choosing this model, so the selection caveat under Code does not apply to them; reworded tasks still might. The screen flagged 34 of the 114 as the same task as, or related to, one in the traces, and on the other 80 this model passes 38 and the first release 26 (16 problems against 4, p = 0.0059); the screen can miss restatements that share few words. GSM8K shows no measurable difference: 881 against 871 of 1,219 (+0.8 points, 95% CI −1.8 to +3.5). With thinking on, both pass 27 of the 50 HumanEval problems, but this model ran out of its 4,096 tokens on 21 of them and the first release on 8, and no answer cut off passed; with 50 problems the interval is ±20 points. The frozen HumanEval 50 and GSM8K 100 with thinking off, and the first 25 HumanEval problems with thinking on, were rescored along the way, with the same result on every problem. The rule's outcome is to replace the first release with this model. The owner decided to publish this version before that result existed, not under the rule; the result agrees with that decision.

Math and thinking

• GSM8K with thinking off is 65/100. That is 4 below the first release, which is within noise (10 problems solved only by this model, 14 only by the first release, p = 0.54). It is 12 below BF16 and Prism's model.
• GSM8K with thinking on went from 25/50 to 32/50 (10 problems against 3, p = 0.09).
• With thinking on, this model often reasons longer than the 4,096-token cap. It ran out mid-thought on 18 of 50 GSM8K problems (BF16 14, Prism's model 4, the first release 23) and on 9 of 25 HumanEval problems (BF16 3, Prism's model 0, the first release 5). All 32 GSM8K answers that finished thinking were correct, so its thinking-on GSM8K losses are answers cut off mid-thought, not wrong answers.
• A second epoch of distillation was tried on both data sets. It did not improve HumanEval: 21/50 against 19/50 on the first release's data (5 problems against 3, p = 0.73), and 29/50 against 34/50 on this data. It made thinking-on answers longer, so more of them were cut off (GSM8K with thinking on, on this data: 32 of 50 answers cut off against 18, and a score of 18/50 against 32/50). On this data it scored higher on GSM8K with thinking off: 74/100 against 65/100 (15 problems against 6, p = 0.08).

General text and Prism's runtime

• It fits general text slightly worse than the first release: dev NLL 3.012 against 2.960, KL 0.607 against 0.554, EAR 0.693 against 0.706, and Prism perplexity 14.15 against 13.61. Its calibration data has half as much WikiText-2 and UltraChat (32 blocks each instead of 64), and its distillation data less UltraChat (512 sequences instead of 640). ARC is 89 against 84.
• Prism's runtime: perplexity on WikiText-2 test, 8 × 512 tokens, with upstream llama.cpp's wiki.test.raw, is 14.15 for this model. The first release scores 13.61, the generic-calibration model 12.88 and Prism's Ternary-Bonsai-2-9B 17.21. Logs: research/runs/2026-10-28-prism-ppl-upstream/.
• Correction: the earlier version of this card gave 12.59 for the first release, 16.51 for Prism's model and 11.93 for the generic-calibration model. Those numbers came from a non-standard copy of the test file, with the blank lines removed and Windows line endings. The upstream file gives the numbers above, and the order of the models is unchanged. That version also gave dev NLL 2.887 and ARC 89 for the generic-calibration model. Those came from a separate re-run of the package, not from the model scored in the rest of that row; that model's own numbers, now in the table, are 2.892 and 87.
• WikiText is partly in-domain. The calibration and distillation data include WikiText-2 train, so the WikiText numbers (dev NLL, KL, EAR and perplexity) favor the OpenBonsai models over Prism's. ARC is not in the data. GSM8K and HumanEval test problems, and trace prompts that share a 13-word run with one, are kept out; reworded HumanEval tasks are not (see Code).
• Non-English output is weak. The calibration data is almost all English. On the one Turkish dev prompt, this version no longer repeats itself as the first release did, but it still produces made-up words.

License and attribution

• Apache-2.0, following the base model Qwen/Qwen3.5-9B.
• The PTQ1_0 layout follows Prism ML's published Ternary Bonsai format.
• The method builds on ScaleQ-1.58 / CAT-Q, GPTQ, EfficientQAT-style distillation and Hadamard-rotation quantization work.
• Calibration prompts come from GSM8K (MIT), MATH (MIT), MBPP (CC-BY-4.0), OpenCodeInstruct (CC-BY-4.0), UltraChat (MIT) and WikiText-2 (CC BY-SA).