Ternary Bonsai 2 27B, NInfer artifact for the RTX 4090

Prism ML's Ternary Bonsai 2 27B converted to NInfer's .ninfer format, with the vision tower and a Qwen3.8 MTP head for speculative decoding. One self-contained file with text, vision and MTP.

These files only run on the NInfer build from JGamboa/ninfer-4090-windows (branch main), on an NVIDIA RTX 4090 (sm_89), on Windows or Linux. They do not load in llama.cpp, vLLM or Transformers.

Files

bonsai2_27b_vl_mtp_q4q5.ninfer and bonsai2_27b_vl.ninfer each have a .conversion.json report next to them: sources, formats and the conversion command. The retrained file is bonsai2_27b_vl_mtp_q4q5.ninfer with only its 13 MTP objects rewritten, in the same stored formats, so that report also describes every other object. SHA-256 of bonsai2_27b_vl_mtp_retrained.ninfer (6,869,420,800 bytes): 4541bf80637933068b1dc5526be87dbb5d77f6eb3cb3bb1ca07356bc4daf6468.

MTP self-distillation (2026-10-11)

Bonsai's MTP layer came from Qwen3.8, trained against the unquantized model. It was retrained against Bonsai's own predictions with tools/mtp_distill: Bonsai answered 3,342 prompts (code in ten languages, English and Spanish chat, edit/JSON/math templates) through ninfer-serve; ninfer-score-export captured its final hidden states and argmax over 3.49 M tokens; one epoch of three chained, teacher-forced draft steps followed.

• Held-out draft accuracy, steps 1 / 2 / 3: 67.8 / 59.7 / 55.3 % → 78.9 / 71.1 / 66.9 %.

• RTX 4090, MTP 3, rk4v4-e8, --lm-head-draft, runs alternated:

  Workload	mtp_q4q5	mtp_retrained
  Six mixed prompts, greedy, thinking off	225.6 tok/s	252.1 tok/s (+11.8 %)
  Four prompts × two seeds, model-card sampling, thinking on, n-gram	190.6 tok/s	220.5 tok/s (+15.7 %)
  JSON and document edits, MTP 3 + n-gram, greedy	611 / 812 tok/s	618 / 840 tok/s

• The main model is unchanged: speculative decoding only accepts tokens the main model verifies. Greedy text can still settle near-ties differently because the verification order changes.

Performance

Measured on one RTX 4090 at stock clocks, Windows 11, CUDA 13.4, with the file bonsai2_27b_vl_mtp_q4q5.ninfer. Current figures (2026-09-27, the 4090 without a display):

Decode by workload (2026-09-24/25, when the card also drove a 4K desktop at 60 Hz, which cost about 15 % of decode), tok/s:

• Quality: 43 of 45 deterministic tasks (code, JSON, tool calling, math, long context, Spanish), against 44 for Qwen3.8-27B.
• Weights in VRAM: 6.11 GiB, or 6.39 GiB with vision.

Every figure, with its method, is in the repository's design notes (section 9.1).

Usage

Download the prebuilt Windows release (its start-bonsai-server.bat uses this file) or build NInfer from the GitHub branch above, then run:

ninfer-serve bonsai2_27b_vl_mtp_retrained.ninfer --host 127.0.0.1 --port 8080 ^
  --max-context 262144 --kv-capacity auto --kv-dtype rk4v4-e8 --max-concurrency 3 ^
  --spec mtp --draft-tokens 3 --lm-head-draft --ngram chain --vision

• The server exposes an OpenAI- and Anthropic-compatible API at http://localhost:8080/v1, and a live monitor at /monitor.
• On a 24 GB card this configuration gives each request the full 262K context. The three lanes share a pool of about 767K KV tokens.
• --ngram chain extends each MTP round with drafts copied from the context. It helps on code editing, JSON, tool calls and documents, and does nothing on free prose.
• The model thinks by default. Prism recommends temperature 1.0, top_p 0.95, top_k 20 in thinking mode.

What's inside

• Ternary weights: Prism's codes are kept bit-exact, repacked as t5_g128_fp16 (5 weights per byte in base 3, one FP16 scale per 128 weights).
• Vision tower: from Prism's Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf.
• MTP head, tokenizer, chat template and configuration: from the official Qwen3.8-27B NInfer artifact. Bonsai keeps the Qwen3.8 architecture and tokenizer. In the recommended file, the MTP layer is requantized to Q4/Q5.

Modifications

This artifact is a derivative of Prism ML's Ternary Bonsai 2 27B. Changes made:

• The ternary weights are repacked from Prism's PTQ1_0 GGUF packing to t5_g128_fp16. The codes are kept bit-exact and the FP16 group scales preserved.
• The vision tower is converted from Prism's mmproj GGUF (Q8_0 dequantized to the NInfer layout).
• The MTP speculative-decoding head, tokenizer, chat template and configuration are added from the Qwen3.8-27B NInfer artifact (derived from Qwen/Qwen3.8-27B). In bonsai2_27b_vl_mtp_q4q5.ninfer, the MTP layer is requantized from Q8 to a Q4/Q5 mix.
• In bonsai2_27b_vl_mtp_retrained.ninfer, the MTP layer's weights are further fine-tuned on Bonsai's own outputs (self-distillation) and stored in the same Q4/Q5 formats.
• Everything is packaged as a single NInfer v3 .ninfer artifact.

License

Apache-2.0, the same license as all three sources: Prism ML's Ternary Bonsai 2 27B, Qwen/Qwen3.8-27B, and neroued/Qwen3.8-27B-NInfer.

Credits

• Model: Prism ML (Ternary Bonsai 2 27B), based on Qwen3.8-27B by the Qwen team.
• Engine: NInfer by Neroued. RTX 3090 and 4090 ports by Don-Chad and sergiuszm; E8 KV by UDPSendToFailed.
• Windows port, Bonsai support, kernels, speculation and conversion: JGamboa, with Claude Code.