Ternary-Bonsai-2-27B — native MTP draft head (GGUF)
A multi-token-prediction (MTP) draft head trained on Bonsai 2's own hidden states, for speculative decoding of prism-ml/Ternary-Bonsai-2-27B-gguf (PQ2_0) in llama.cpp. The head proposes up to 3 tokens; the 27B model verifies them in one pass, so accepted drafts come almost free.
| file | size | sha256 |
|---|---|---|
mtp-bonsai2-native-v5b-dv64k.gguf |
1,055,253,344 bytes | 9d11de22b83258b1b16ea7c73b0148d9c38a96f0cc0c077c55b9ce676ba287e7 |
Requires the Pascal patch build (GitHub). PrismML's stock fork refuses
to load this head: prism.hadamard: weight 'blk.64.nextn.shared_head_head.weight' is not on a verified Hadamard-aware matmul path.
Speed (GTX 1080 Ti 11 GB)
| 32K context, f16 KV | with this head | without |
|---|---|---|
| code (chat prompts, greedy, 300 tokens) | 86.4 tok/s | 42.6 (plain decode) |
| prose | 64.3 | |
| math | 98.4 | |
| a document summary, 2K of the context filled | 75.0 | 41.7 |
| a document summary, 30K of the 32K context filled | 53.4 | 34.1 |
Conditions: no display on the card, memory +1000 MHz (6003 MHz under CUDA load), one request at a time, the Pascal patch build. Full method and caveats in that repo's README.
Use
set GGML_CUDA_ROWLANE=1
set GGML_CUDA_RL_N4_LONG=1
llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -c 32768 -fa on -np 1 --spec-type draft-mtp -md mtp-bonsai2-native-v5b-dv64k.gguf -ngld 99 --spec-draft-n-max 3 --spec-draft-p-min 0.4 --backend-sampling --jinja
--spec-draft-n-max 3 and --spec-draft-p-min 0.4 are the tuned values (deeper drafts cost more to verify than they
return on this card). Send reasoning_effort: "medium": Bonsai 2's chat template defaults to xhigh and returns
HTTP 500 for "high".
What it is
- One transformer block (
blk.64) plus Bonsai 2's own output head, with the Hadamard rotation folded in and the output restricted to a 64K-token draft vocabulary (pruned from 248,320) so drafting stays cheap. - Trained on Bonsai 2's own generations (on-policy), with the exact inference-time masking; the final round added 8,949 answers / 5.4M tokens.
- Held-out agreement with the 27B's next tokens: 0.910 / 0.794 / 0.740 at draft depths 1 / 2 / 3. Live acceptance is content-dependent: ~0.77 on code, ~0.57 on prose.
Credits
Base model © PrismML, Apache-2.0; this head is a derivative and is released under the same license. Training, export and measurements by Claude (Anthropic) for @killy-netsphere.