Ternary Bonsai 2 27B: OpenVINO INT2 for Intel Arc (community conversion)

An unofficial OpenVINO conversion of PrismML's Ternary Bonsai 2 27B. Bonsai 2 is a ternary version of Qwen3.8 27B.

This version (v2, "Arc build") is tuned for decode speed on Intel Arc A-series: ~29 tok/s on one Arc A770 16 GB (greedy, batch 1), up from ~16.9 tok/s in v1, with the same weights. It needs a patched OpenVINO GPU plugin, included in v2-arc/plugin/ (Windows). The previous version (v1) runs without the patch and is still here, in v1-portable/.

Versions in this repo (one folder each)

Download one folder (the whole repo is ~17 GB):

hf download TheSanchezian/Ternary-Bonsai-2-27B-int2-ov --include "v2-arc/*" --local-dir bonsai2
hf download TheSanchezian/Ternary-Bonsai-2-27B-int2-ov --include "v1-portable/*" --local-dir bonsai2

The model folder is then bonsai2/v2-arc (or bonsai2/v1-portable). v1 is also available as the git tag v1-portable.

The ternary weights are unchanged and lossless. PrismML's PQ2_0 weights (2-bit codes, value = code − 1, one fp16 scale per 128 weights) are stored directly as OpenVINO u2 weights (the layout NNCF uses for INT2_SYM, group size 128). Nothing is re-quantized. v2 changes only how the graph and the GPU kernels compute.

What changed from v1

Requirements (important)

• Intel Arc A-series GPU, Windows. Tested on an Arc A770 16 GB. Other GPUs, Linux and the CPU are untested.
• OpenVINO from PR #38837 (CI build 2026.4.1-63022, it has working u2 GPU kernels) with OpenVINO GenAI 2026.4.1: install the PR's openvino wheel, then pip install --no-deps openvino-genai==2026.4.1 openvino-tokenizers==2026.4.1 so the PR's openvino stays in place.
• The patched GPU plugin: copy v2-arc/plugin/openvino_intel_gpu_plugin.dll over the file of the same name in site-packages/openvino/libs/ of that install (keep a backup of the original). Without it the model does not load (the custom ops need it). The full source change is v2-arc/plugin/bonsai_plugin.diff against the commit in v2-arc/plugin/BASE.txt; Linux users can build the plugin from it (untested).
• Python packages: numpy, pillow (images).

Usage

python bonsai2/v2-arc/runtime/run_bonsai.py bonsai2/v2-arc "What year did the Berlin Wall fall, and why?"
python bonsai2/v2-arc/runtime/run_bonsai.py bonsai2/v2-arc --image picture.png "Describe this image."
python bonsai2/v2-arc/runtime/run_bonsai.py bonsai2/v2-arc --think "Solve: ..."

In your own code, register the custom ops before creating the pipeline:

import sys; sys.path.insert(0, "bonsai2/v2-arc/runtime")
import bonsai_customops, openvino_genai as ov_genai
ext, cfg = bonsai_customops.genai_props("bonsai2/v2-arc")
pipe = ov_genai.VLMPipeline("bonsai2/v2-arc", "GPU", extensions=ext, CONFIG_FILE=cfg)

Prompt format: Qwen3.8 chat format; <think>\n\n</think>\n\n after the assistant tag turns thinking off, <think>\n turns it on (see v2-arc/runtime/run_bonsai.py). Switches (environment, all optional): OV_GPU_BONSAI_GEMV2=0 stock u2 kernel; BONSAI_AB_V2=0 previous GDN gate kernel.

Results (Intel Arc A770 16 GB, Windows, GPU alone)

Setup: OpenVINO GenAI VLMPipeline, greedy, batch 1, thinking off unless stated. The second GPU in the PC was idle during all measurements.

Long context (a needle test: one fact hidden mid-document, thinking off):

(The 2k row includes the first long prompt's warm-up.) At long context, writing runs at ~20–22 tok/s: the 16 full attention layers read a growing KV cache, while the 48 linear-attention layers keep a fixed-size state.

Not measured: standard benchmarks; speculative decoding (MTP/PLD) on this version.

How it was made

1. Conversion (as in v1): v2-arc/conversion/convert_bonsai2_ov.py maps every Bonsai weight onto Intel's OpenVINO/Qwen3.8-27B-int8-ov skeleton (details below in the v1 section), then folds the rotation signs and fuses horizontally (qkv+z, a+b, q+k+v, gate+up).
2. Graph passes (new in v2): the rotation runs on rank-3 tensors, and the small ops around each matmul are fused into custom GPU ops (OpenCL): RMSNorm + gamma + signs + block Hadamard (RmsRot), SwiGLU / sigmoid gate + rotation (GluRot, SigRot), GDN a/b projections with softplus/sigmoid (AbG, AbB), GDN output norm + gate + rotation (GdnNR). Every pass and every op was checked against the unfused graph (CPU, f32). The full chain with commands is in v2-arc/conversion/README.md.
3. Plugin (new in v2): custom ops without per-shape recompiles, fewer allocations per token, and a new u2 decode kernel; see v2-arc/plugin/NOTICE.md. These keep OpenVINO GenAI's paged-attention pipeline intact.
4. Answers were compared with the previous builds on a fixed question set after every step: same facts, numbers and code.

Previous version (v1)

v1 is in the folder v1-portable/ (its own README is there too). It runs on the plain OpenVINO PR #38837 build, without the patched plugin (~16.9 tok/s on the A770):

hf download TheSanchezian/Ternary-Bonsai-2-27B-int2-ov --include "v1-portable/*" --local-dir bonsai2

Its notes, unchanged (the files they mention, e.g. conversion/ and tools/, are in v1-portable/; the experimental kernel work below grew into v2's decode kernel):

What's inside (v1)

About the rotation: Bonsai 2 stores most weights in a rotated basis. Prism's block Hadamard transform (block 1024, explicit ±1 signs) is applied to the activations at runtime, as in PrismML's llama.cpp fork. This conversion puts that transform into the OpenVINO graph as extra MatMuls in front of each rotated weight.

Requirements (v1)

• An OpenVINO build with working u2 (INT2) GPU kernels.
  • Tested: the CI build from OpenVINO PR #38837 (2026.4.1-63022) with OpenVINO GenAI 2026.4.1.
  • On the official 2026.4.1 release, u2 weights compute incorrectly on Arc A-series GPUs. Expect support in a later official release.
• VLMPipeline from OpenVINO GenAI (Qwen3.8 model type).
  • With the PR build: install the PR's openvino wheel, then pip install --no-deps openvino-genai==2026.4.1 openvino-tokenizers==2026.4.1, so the PR's openvino stays in place.
  • With ContinuousBatchingPipeline: set SchedulerConfig.max_num_seqs small (e.g. 2). The linear-attention state (about 150 MB per sequence slot) is reserved per slot, and the default of 256 slots doesn't fit.
• The CPU device is untested.

Results (v1, Intel Arc A770 16 GB, Windows)

Setup: thinking off, greedy decoding, max 300 new tokens.

For reference, on the same GPU: Gemma 4 26B-A4B int4 (OpenVINO) uses about 14.1 GB at 29–37 tok/s. Gemma is a mixture-of-experts model with about 4B parameters active per token. Bonsai 2 is dense (27B active), so it's slower per token, but much smaller.

Long context (a needle test: one fact hidden mid-document, thinking off):

Reading speed stays flat with length, and writing barely slows: 48 of the 64 layers are linear attention with a fixed-size state.

Speculative decoding on this build (A770, batch 1):

• MTP with Qwen3.8's MTP head works (up to 97% acceptance), but it's slower than plain decode (0.63–0.79x): the current u2 kernels are much slower for multi-token steps (M = 2–6) than for M=1.
• Prompt-lookup decoding helps only on near-pure copying (1.57x on a rename-style code edit; 0.41x on chat).
• Plain decoding is recommended until u2 small-M kernels improve.

Not yet measured: standard benchmarks.

Experimental kernel (v1 notes)

The bottleneck: on Arc A770 the u2 FullyConnected path in OpenVINO reaches ~115 GB/s effective. It also switches from gemv to bf_tiled as soon as a step has 2+ tokens, which doubles the step time and makes speculative decoding a loss.

The benchmark: a standalone OpenCL kernel (tools/u2_kernel_bench.py, pyopencl, results checked against numpy) using an interleaved weight layout:

• M=1: 268–307 GB/s on Bonsai's layer shapes, vs a measured peak read of ~434 GB/s.
• M=2: only 3–11% slower than M=1.

Plugging it into OpenVINO: tools/ternary_op.py packages it as an OpenVINO GPU custom op (TernaryFC_<N>_<K>, loaded via CONFIG_FILE); a single layer runs correctly inside OpenVINO with a dynamic sequence length. Full-model results are in progress; this repo's model still uses the standard u2 path.

Usage (v1)

import openvino_genai as ov_genai
pipe = ov_genai.VLMPipeline("<this repo, downloaded>", "GPU")
cfg = ov_genai.GenerationConfig(); cfg.max_new_tokens = 300; cfg.apply_chat_template = False
prompt = "<|im_start|>user\nWhat year did the Berlin Wall fall?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
print(pipe.generate(prompt, generation_config=cfg).texts[0])

• That prompt turns thinking off (the empty think block).
• For thinking on, end the prompt with <think>\n instead.

How it was made (v1)

1. The skeleton is Intel's OpenVINO/Qwen3.8-27B-int8-ov. The converter replaces every weight with Bonsai's:
   • the 400 rotated linear weights + lm_head: u2, plus the Hadamard transform in the graph;
   • the GDN small projections: BF16 → f32;
   • conv1d, A, dt_bias and all RMSNorm weights: copied, undoing llama.cpp's GGUF transforms (V-head reordering, ssm_a = -exp(A_log), norms stored as 1 + w);
   • the embeddings: un-rotated (s ⊙ H z), then int8 per row.
2. Every replaced tensor was checked against the original Qwen3.8 weights (correlation of the effective weights).
   • Median 0.884 for the big ternary weights; 0.98–1.00 for the norms; 1.000 for A/dt_bias.
   • A wrong mapping shows up near 0.
3. The converter passed a synthetic end-to-end test: max relative error about 1e-6 against an independent reference.
4. Not included: Qwen's MTP head. Bonsai 2 ships no MTP layer.

The converter script is included in this repo under conversion/ (convert_bonsai2_ov.py, with its synthetic test). Run it with check for a read-only mapping report before write.

Credits and license

• Model: PrismML, Ternary Bonsai 2 27B, Apache 2.0. Based on Qwen3.8 27B (Alibaba Qwen team).
• OpenVINO skeleton: Intel's OpenVINO Qwen3.8-27B release.
• Plugin: built from OpenVINO (Apache 2.0) with the changes in v2-arc/plugin/bonsai_plugin.diff; see v2-arc/plugin/NOTICE.md.
• This repo is an independent format conversion. It's not affiliated with or endorsed by PrismML, Alibaba or Intel.