Bonsai 2 27B + MTP (GGUF)

prism-ml/Ternary-Bonsai-2-27B-gguf (PQ2_0) with the multi-token-prediction head from Qwen3.8-27B grafted in, so it can drive --spec-type draft-mtp speculative decoding.

~44% faster decode at the same quality — the ternary weights are untouched, and speculative decoding verifies every draft against the target model, so the output distribution is unchanged.

Measured on one NVIDIA RTX PRO 6000 Blackwell MIG 1g.24gb slice (a quarter GPU, 24 GB), driver 580.126.20, --parallel 1, 400 generated tokens, 5 prompts, median. On a full GPU the absolute numbers will be higher; the ratio is the durable part.

---

⚠️ Requires a patched llama.cpp

With stock binaries this model loads and runs normally, but --spec-type draft-mtp fails at startup:

llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM
llama_init_from_model: failed to initialize the context:
    Hadamard-latent table 'token_embd.weight' is read without the inverse transform

This is not a defect in the graft. PrismML's ternary models store token_embd.weight in a rotated ("Hadamard-latent") domain, and every read must apply the inverse transform first. The main model path in PrismML-Eng/llama.cpp does this; the MTP draft graph, which is upstream llama.cpp code predating the folding, has its own embedding lookup and skips it.

0001-qwen35-mtp-hadamard-inverse.patch in this repo fixes it in ~8 lines against tag prism-b10685-7dffb15. Build:

curl -fsSL -o src.tgz https://github.com/PrismML-Eng/llama.cpp/archive/refs/tags/prism-b10685-7dffb15.tar.gz
mkdir src && tar -xzf src.tgz -C src --strip-components=1
cd src && patch -p1 < ../0001-qwen35-mtp-hadamard-inverse.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=<your arch> -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

Single CUDA arch, server target only: ~150 s on 16 cores.

Run

llama-server -m Bonsai-2-27B-PQ2_0-MTP.gguf \
  -ngl 99 -fa on -c 32768 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2

Sampling, per the base model card — thinking mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0; instruct: temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.

--spec-draft-n-max 2 was best on a 24 GB slice. Sweep it for your card; larger values draft deeper but acceptance falls.

Concurrency

Unusually for speculative decoding, the gain here does not vanish under load — the MTP head is one small block, so drafting is cheap against a bandwidth-bound ternary trunk:

(both -c 65536 -np 4.)

There is a reproducible dip at exactly 2 concurrent streams, where per-request throughput falls to 29 tok/s — below the 37 tok/s seen at 3 concurrent — and MTP ends up ~7% behind vanilla. Confirmed over four runs. Everywhere else MTP leads by 31-54%. If your workload sits at exactly 2 concurrent requests, benchmark before adopting.

The MTP draft context also costs VRAM, so on a 24 GB card you can allocate fewer slots than vanilla: vanilla reaches -np 32 and peaks at ~186 agg tok/s, while this model OOMs past -np 16 and peaks at ~175. Vanilla wins the absolute aggregate ceiling by ~7%; MTP wins everywhere below it.

How it was made

15 blk.64.* tensors — a full transformer block plus the nextn projections — were copied verbatim and unrotated from unsloth/Qwen3.8-27B-GGUF UD-Q2_K_XL into the Bonsai GGUF, with qwen35.block_count set to 65 and qwen35.nextn_predict_layers to 1.

No rotation was applied to them, and they are deliberately not listed in prism.hadamard.weight_names. The justification: Bonsai's RMSNorm weights match stock Qwen3.8 elementwise (cosine > 0.99996 on blk.0.attn_norm, blk.0.post_attention_norm, output_norm). An elementwise scale cannot commute with a rotation, so the residual stream is in the original basis and prism.hadamard.* is an online per-matmul quantisation aid, not a change of basis. A stock MTP block therefore receives exactly the activations it expects.

Only the embedding table is genuinely latent, which is why the one patch is needed.

Base tensors are bit-identical to the source GGUF; only metadata changed and 15 tensors were appended. Merge tool: graft_mtp.py in this repo.

Acceptance

0.60 here versus 0.70 for the same head on stock Qwen3.8 — the expected cost of a head trained on FP16 hidden states drafting for a ternary trunk. Well above break-even, hence the speedup.

Provenance and attribution

Not affiliated with, or endorsed by, PrismML or the Qwen team. All components are Apache-2.0, as is this derivative.

• Language model: prism-ml/Ternary-Bonsai-2-27B-gguf (Apache-2.0)
• MTP head: unsloth/Qwen3.8-27B-GGUF UD-Q2_K_XL (Apache-2.0), quantised from Qwen/Qwen3.8-27B (Apache-2.0)
• Runtime: PrismML-Eng/llama.cpp prism-b10685-7dffb15 (MIT)
• MTP recipe and the observation that the head ships inside the GGUF: sudoingX/qwen38-mtp

Limitations

• Needs the patch for MTP. Without it, works as a normal Bonsai 2 27B.
• Quality was not re-benchmarked. The claim that quality is unchanged rests on speculative decoding being verified-lossless and the base weights being bit-identical — not on a benchmark run.
• The MTP block is Q6_K/Q8_0, not ternary, adding ~0.35 GB over the base model.
• Tested only on CUDA sm_120. Metal and CPU paths are untested.
• The vision mmproj is not included; text-only.