Bonsai 2 27B, served: the full 262,144-token window at q8_0 on a 12 GB card

A 27B model with its whole trained context window, at q8_0 KV precision, on one RTX 4070 12 GB. 78 tok/s of new text at 32k tokens of context (175 on code), 69 at 64k, 61 at 112k, and the window keeps going to 262k. Lossless speculative decoding at every depth. A server that answers the requests agents and apps actually send. One command on Windows.

This is PrismML's Ternary Bonsai 2 27B (1.58-bit ternary PTQ1_0, 5.9 GB, Apache 2.0), the same weights, nothing re-quantized, with its MTP draft head grafted back on (byte-identical trunk, proof below) and the serving stack from github.com/professorpalmer/bonsai-ada-surgery: a tiered KV cache that puts the first ~119k positions in VRAM and the rest in pinned system RAM with bit-identical output, MTP drafting that pays at every depth, harness-proofing for the apps that send effort: "high" or tiny output caps, and an optional server-side layer (exact API cards, an API check, a sandboxed Python tool) that lifts the model's coding and agentic scores without touching the weights.

Measured 2026-10-08 (bench/depth_table.sh), a fresh prompt at each depth: 400-token greedy answers with the default drafting (lookup + MTP head); new text = an essay, code = a Python module; prefill = cumulative for the whole prompt; display on the CPU's iGPU. Every number has a receipt in the repository (docs/Q8_FULL_CONTEXT.md, docs/RECEIPTS.md, docs/QUALITY.md).

What the serve changes, same weights

Most of the quality gap people report on Bonsai 2 for code and agents is runtime, not compression: the effort word the template rejects, the output cap that ends a thinking model mid-thought, the tool-call format. The server fixes those; the layer adds exact references and a sandbox. The weights are untouched.

Files

The original Ternary-Bonsai-2-27B-PTQ1_0.gguf without the head also works with everything here (BONSAI_SPEC=0 for no drafting, or let the launcher fall back); get it from prism-ml/Ternary-Bonsai-2-27B-gguf.

Run it (Windows, NVIDIA RTX 20 / 30 / 40 / 50)

git clone https://github.com/professorpalmer/bonsai-ada-surgery && cd bonsai-ada-surgery
hf download CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF bonsai-bundle-win-x64.zip --local-dir .
Expand-Archive bonsai-bundle-win-x64.zip -DestinationPath . -Force
hf download CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF Ternary-Bonsai-2-27B-PTQ1_0-mtp-procreations.gguf --local-dir models
.\start-server.ps1

OpenAI-compatible API on http://<host>:8080/v1, bearer key in artifacts\api_key.txt (created on first run), llama.cpp's chat UI on the same port. The launcher prints the VRAM line it chose. Optional, once: layer\fetch_runtime.ps1 downloads the sandbox runtime for the layer (CPython 3.12 on WASI, checksummed, 14 isolation canaries); without it the plain server runs.

Linux: bash build/build_linux.sh in the repository builds the pinned PrismML source with all 66 patches applied; the full-window command line is in the repository README. No prebuilt Linux binary yet.

Cards: 12 GB is the measured recipe: q4_0 K/V by default from bundle-20261016, 260,352 of 262,144 positions in VRAM with the display on the iGPU, 242,176 with the display on the card (copies, needles and HumanEval at 245k equal to q8_0; BONSAI_CTK=q8_0 for the previous cache). 16 GB and up: the whole q8_0 window fits, the tier switches itself off. 8 GB: the launcher detects the card and uses its 8 GB preset (q3_0 K/V since bundle-20261015b, q4_0 before; a 262k window with 24 GB of RAM, else 131k; a Q4_0 MTP head; from bundle-20261016b prefill attention packs 2 query heads per column group on Turing: cold 245k 1.147x on an RTX 2060 SUPER, perplexity +0.02 %; from bundle-20261017 saved conversations (BONSAI_SLOT_SAVE_PATH, automatic with the layer) restore a 199k conversation ~6 s after a restart on this card; from bundle-20261017b the launcher warns with a spill line when another program fills the card after the start, which on this card halves reads or worse). Measured on an RTX 2060 SUPER that also draws the desktop: 51.2 / 43.1 tok/s at 4k / 32k (q4_0); with q3_0, 80,640 positions in VRAM, 64k 41.0, 128k 22.8, 192k 12.3 tok/s, HumanEval medium 163 of 164 (repository docs/8GB.md, docs/research/release-20261014b/).

For agents and apps

• Send tools and let the server format and parse the calls (native Qwen3-Coder XML, grammar-constrained).
• Tool-heavy agents: chat_template_kwargs: {"enable_thinking": false} per request. Chat and coding answers: the server default (medium reasoning, 20k thinking budget, forced close).
• Clients that cap max_tokens low are fine: with thinking on the server raises the cap.
• effort: "high", "xhigh", "low": mapped to medium, answered, logged. BONSAI_HARNESS_PROOF=0 passes words through unchanged.

Knobs

BONSAI_CTX (262144), BONSAI_CTK (q8_0), BONSAI_TIER (1), BONSAI_KV_VRAM_CELLS (auto), BONSAI_VRAM_MARGIN (1000 headless / 1300 with the display on the card), BONSAI_SPEC / BONSAI_SPEC_DEEP (2 / 4), BONSAI_DRAFT_WINDOW (16384), BONSAI_EFFORT (medium), BONSAI_THINK (1), BONSAI_THINK_BUDGET (20480), BONSAI_HARNESS_PROOF (1), BONSAI_LAYER (1), BONSAI_PORT, BONSAI_MODEL, BONSAI_MMPROJ / BONSAI_MMPROJ_GPU (vision projector; image encoder on the CPU by default, no VRAM cost). Full table and the VRAM arithmetic in the repository README.

How it works, in one paragraph each

Tiered KV cache (--kv-vram-cells). The first N positions of every layer's K/V live in VRAM, the rest in pinned system RAM mapped into the same CUDA address range. Kernels are unchanged and output is bit-identical to an all-VRAM cache. Past the line, attention copies the used RAM rows into VRAM with the copy engine first, so decode there is bound by PCIe (4.0 x16 here). The box needs ~8 GB of free RAM for the tail at 262k.

MTP drafting at every depth (--spec-draft-window). The draft head only needs recent context, so its cache keeps the last 16k rows and stays small. With quantized-KV decode on the tensor-core attention kernel (one K/V read per verify batch), drafting pays at every depth; greedy output equals greedy output without the head.

Harness-proofing (--reasoning-effort-allow, --reasoning-max-tokens-floor). The same weights score 0 to 160 of 164 on HumanEval depending on what the client sends. The server absorbs both causes.

Lookup drafting (patch 0039, on by default). An n-gram drafter drafts text that is already in the context, up to 32 tokens at a time, in front of the MTP head. File rewrites 117 -> 352 tok/s at 4k and 18.5 -> 66 at 250k; new text unchanged; same output.

Prefill past the VRAM line (patch 0041). The K/V rows a prefill batch writes past the line go to VRAM first and then to system RAM in whole blocks; the next layer's RAM rows are copied while the layers before it compute. A 258k prompt: 369 -> 436 tok/s, same output.

Many tools cost no decode speed (patch 0042). With speculative decoding the server copies the sampler on each draft step; with tools, the copy of the tool-call grammar searched every grammar element, about 15 ms per step with 107 tool schemas. Now a binary search: 107 tools at 63k, 41.8 -> 61.2 tok/s (2 tools: 63.5), same output.

The layer. A small proxy the launcher starts in front of llama-server on the same port. Exact API cards for the Python modules a request involves (generated from the sandbox's own runtime), an API check that flags names that do not exist, and a sandboxed Python tool with the user's text as input.txt. Clients change nothing.

Proof the weights are untouched

Checked tensor by tensor against PrismML's published file (bench/gguf_tensor_identity.py in the repository, run 2026-10-06): all 851 tensors of Ternary-Bonsai-2-27B-PTQ1_0.gguf are byte-identical in this file; the only tensors added are the 15 of blk.64.*, the draft head. Metadata differs by five added keys (graft.*, qwen35.nextn_predict_layers, block count 64 -> 65) and by the embedded chat template, which is this repository's template: PrismML's, plus a system block of operating rules for tool use (save long command output to a file and read the part you need, and so on) that every measurement here was taken with. The launcher uses the GGUF's embedded template with --jinja, so this file serves with that template; PrismML's original file serves with PrismML's.

Every kernel in the stack is checked against the CPU reference, and the served output is checked greedy token-for-token against the unpatched fork before any speed number is recorded.

Credits and licenses

• PrismML for Bonsai 2 27B and the llama.cpp fork this runtime is built on. The serving patches are submitted there (#319-#323, #295, #296, #333; earlier work merged in #214, #216, #221).
• sudoingX for the planar-transposed activation layout, the batch-invariant mode, the MTP graft tools and the Hadamard fix for the draft graph (bonsai2-small-gpu, PrismML #217, #218).
• ProCreations for the on-policy MTP draft head (Ternary-Bonsai-2-27B-MTP).
• Alibaba Qwen for Qwen3.5-27B, the base model.
• Testers who filed issues with full logs: @Milor123 (empty JSON-schema unions, the vision projector and GPU sharing, deep-context agent runs) and @outbackdingo (GCC 16 build).
• The serving stack, launcher, layer, suite and docs: MIT, Cary Palmer. The weights are Apache 2.0 as published by PrismML; this repository redistributes them with the head added and the notices kept. Not affiliated with, endorsed by or sponsored by PrismML, sudoingX, ProCreations or Alibaba.

Issues and measurements: github.com/professorpalmer/bonsai-ada-surgery.