Ternary Bonsai 2 27B, NInfer v3 artifact

A NInfer v3 artifact of PrismML's Ternary Bonsai 2 27B for NInfer-all, the master branch of iamwavecut/ninfer-all, which serves the RTX 3090, RTX 4090 and RTX 5090. Stock NInfer builds refuse this file (t2_g128_fp16 and the hadamard_signs Use auxiliary exist only in that line).

The numbers below come from master at 2172a598 (September 2026), each card running the device route profile built into it. Every profile of this revision runs there; older engines are covered at the end.

What is inside

With --lm-head-draft the drafters score their candidates on the proposal head instead of the full one. MTP decodes 4% faster through it and accepts the same share of its drafts; DFlash2 gains nothing measurable, so its profiles leave the flag off. The head is loaded only with the flag.

One .ninfer file of 9,520,051,456 bytes (8.87 GiB), next to its conversion report, SHA256SUMS and NOTICE.

Conversion command, from the repository's tree (add --device cuda to encode the Q8 parts on a GPU):

python3 -m tools.convert \
  --model Qwen3.8-27B \
  --recipe bonsai2_27b_ternary \
  --source ternary=Ternary-Bonsai-2-27B-PQ2_0.gguf \
  --source mtp=Ternary-Bonsai-2-27B-MTP/model_mtp.safetensors \
  --source dflash2=Ternary-Bonsai-2-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --proposal \
  --name bonsai2-27b \
  --out Ternary-Bonsai-2-27B-ninfer-v3.ninfer

Running

Build master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a). The all-round single stream, over the model's whole 262,144-token window:

ninfer-serve Ternary-Bonsai-2-27B-ninfer-v3.ninfer --model-id bonsai2-27b \
  --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec dflash2 --draft-tokens 5

• Seven drafts are faster on short answers; after long documents the best count lies between three and seven.
• --spec mtp --draft-tokens 3 --lm-head-draft drafts with the MTP head through the proposal head.
• --vision --vision-residency overlay --vision-max-merged 12288 adds images: the tower waits in pinned host memory, and an encode borrows device memory from the output head, the token table and the loaded drafter, which are restored afterwards. Without a drafter, overlay takes --vision-max-merged 8192 (or --vision-residency resident).
• The RTX 3090, 4090 and 5090 start with their measured route profile built in; any other GPU is calibrated once at first start (20 to 40 seconds) and keeps the result.

Device memory of the idle server at the 262,144-token window, one lane, text only (RTX 3090; the RTX 4090 and 5090 differ by at most 0.2 GiB):

At a 198,400-token window the MTP head costs 0.73 GiB (0.42 GiB of weights plus the KV and graphs of its attention layer), the proposal head another 0.16 GiB when --lm-head-draft loads it, the DFlash2 adapter 1.48 GiB, and the Vision tower 0.73 GiB resident or 0.03 GiB in overlay.

Largest context

The largest window each card starts with (--max-context = --kv-capacity, one request, --rope-yarn past 262,144), without speculation / with MTP (3 drafts) / with DFlash2 (5 drafts); 1,048,576 is the engine's ceiling:

Filled to the largest window without speculation, with three codes planted at 33, 66 and 90% of the document and asked back in order, the model returned all three on every card up to 978,944 tokens (RTX 3090 with rk4v4 at 970,752: prompt in 2,359 s, then 17.7 tok/s; RTX 4090 at 958,464: 991 s, 30.9 tok/s). At 1,048,576 tokens, which only the RTX 5090 holds, it misses the code at 90% (about 943K), with --rope-yarn and with plain RoPE alike.

ninfer-serve Ternary-Bonsai-2-27B-ninfer-v3.ninfer --model-id bonsai2-27b \
  --max-context 958464 --kv-capacity 958464 --kv-dtype rk4v4 --gdn-state-fp16 --rope-yarn

Results

September 2026, rk8v4 KV, 262,144-token window, one request at a time, greedy, thinking off; RTX 3090 at 390 W (the seven-draft row on a 370 W board), RTX 4090 at 450 W, RTX 5090 at 575 W. Short chat is the mean decode rate over five 512-token answers (an essay, a Python module, a word problem, a Russian explanation, a list); at depth, a ~400-word answer after a document of that length.

With speculation the decode rate at depth follows the drafts accepted on that one answer, so it moves from depth to depth. Eight requests at once run eight lanes of 16,384 tokens; there MTP holds its lead over no speculation (1,053 tok/s on the RTX 5090) while DFlash2 falls behind it (799).

Against the previous master on the same host, a 261K-token prompt now takes 215 s instead of 315 s on the RTX 3090, 102 s instead of 138 s on the RTX 4090 and 82 s instead of 115 s on the RTX 5090, and decode after it with rk4v4 is 11% faster on the RTX 3090 and 13% on the RTX 4090. Decode on short prompts is unchanged: it is bound by reading the ternary weights.

Every table, KV coding (rk4v4, rk4v4-e8), draft length from one to fifteen and the method are in the reference measurements.

Quality

On a fixed 1,179-item slice, greedy decoding with no sampling penalties, this revision on the engine at 897f89a3 against the previous revision on its engine:

GSM8K scores 95.5 under MTP and 96.5 under DFlash2 against 94.5 for llama.cpp on the source GGUF. Perplexity (ninfer-perplexity --quick, ninfer-ppl-1m-v1, 4,096-token windows at a 2,048 stride, rk8v4) is 5.630: 7.820 WikiText, 8.663 PG-19, 7.903 Chinese Wikipedia, 1.861 code. The dense Qwen3.8-27B NInfer artifact measures 4.346 under the same settings. On the current master, the fast prompt kernel moves perplexity at 64K from 5.2074 to 5.2079 with rk8v4 and from 5.2192 to 5.2200 with rk4v4.

Retrieval: three codes planted at 33, 66 and 90% of a 261K-token document are all returned, in order, on the RTX 3090, 4090 and 5090 with rk8v4, rk4v4 and rk4v4-e8. Four 60,000-token documents served at once return all twelve codes.

A request served beside others can come out in other words than the same request served alone: the device profile picks each width's fastest schedule, and a near-tied token can then flip. --device-profile off keeps the previous master's compiled schedules. See batch composition.

Older engines

The line used to be the branch franken/v0.11, whose history is kept under the tag franken-v0.11. There this revision needs 3fb3d33e or later, and DFlash2 through its proposal head (--spec dflash2 --lm-head-draft) needs ce02e2cf or later. Its DFlash2 adapter's output and down projections are Q4 and reach the residual through a path older builds lack: builds from 1e9e01e1 on load the text, Vision and MTP profiles but refuse --spec dflash2 at startup with q4 linear: unsupported shape. On those builds the previous revision, 8beb07b, runs every profile. Builds older than 1e9e01e1 refuse both at startup with embedding: unsupported table qtype.

Changes from the previous revision

• Proposal head. The output head's own ternary rows for the 131,072 most frequent tokens, copied without re-encoding, back --lm-head-draft: MTP decodes 4% faster through it than through the full head and accepts the same share of its drafts (70.4% against 70.0%). It adds 170 MiB to the file and 0.16 GiB of device memory, only when the flag is on. The previous card advised against a proposal head: the one tried then was re-encoded rather than copied from the ternary head, and it lowered the Bonsai-trained MTP head's acceptance.
• Engine (897f89a3). The producers that write the rotated projection inputs run each 1024-point Hadamard transform on four warps instead of one, bit-identical, and the output, draft and proposal heads take the int8-activation route of the text projections: decode is 4 to 5% faster in every profile. DFlash2 now runs through a ternary proposal head as well.

Credits and license

Ternary weights: PrismML, Ternary Bonsai 2 27B, derived from Qwen/Qwen3.8-27B. MTP head and DFlash2 adapter: ProCreations, trained on Bonsai 2 from Qwen3.8-27B's MTP head and z-lab's Qwen3.8-27B-DFlash2. Vision and the tokenizer: Qwen/Qwen3.8-27B. All are Apache-2.0; their attribution notices are collected in NOTICE. This repository only re-encodes those weights into NInfer's container.