Ternary Bonsai 2 27B, with a draft head of its own

Prism ML's Bonsai 2 27B -- a 27B model with ternary weights, 8.6 GB on disk -- together with a multi-token prediction head that GreenBit AI trained for it, in mtp/. With gbx-lm, speculative decoding works from this repository alone.

On a Mac Studio it generates at 114 tok/s on short prompts -- 2.7x Prism's own MLX runtime, 2.6x llama.cpp and 1.7x the fastest other speculative decoder for this model -- and says the same thing. On an NVIDIA DGX Spark it generates at 62 tok/s, 2.1x llama.cpp's CUDA build, reads long prompts 1.6x as fast (a 16k-token document's first token in 10.5 s instead of 16.6 s), and holds the model in 5.9 GB of memory after loading. Every token the head proposes is checked by the model itself, so the reply is the model's own; the head only saves passes over the weights.

The weights are Prism ML's, unchanged, byte for byte. Bonsai 2 ships without a draft model; this head starts from the multi-token prediction head of Qwen3.8-27B, which Bonsai 2 is built from, and is distilled on text generated by Bonsai 2 itself -- prose, code, chat and coding-agent sessions.

Speed

Against the other runtimes for this model

Measured 2026-10-20 on a 512 GB Mac Studio (M3 Ultra), every runtime on the same machine and the same prompts: two each of code, prose and chat, and a ~4k- and a ~16k-token document to summarize. Greedy, thinking off, up to 512 tokens a reply, prompt cache off, the median of three runs after a warm-up; decode is timed from the first token. All runtimes produced the same text.

Decode speed in tokens per second; in brackets, how many times as fast gbx-lm with this draft head is. "short, all" is the mean of the six short prompts. Versions: gbx-lm 0.8.2; MTPLX 2.12.2 with Youssofal/Ternary-Bonsai-2-27B-MTPLX-Optimized-Speed; llama.cpp Prism fork 73bfe6c (Metal, flash attention on; the PTQ1_0 GGUF is slower here, 42 tok/s); Prism's runtime from the MLX pack with mlx 0.32.0.

Prompt processing

gbx-lm reads long prompts the fastest of the four, 1.2–1.4× as fast as Prism's own runtime:

Tokens per second, the same documents and runs as above. On Apple GPUs the 27B's arithmetic, not its weights, sets the pace of prompt processing, so the runtimes sit close together here; on the DGX Spark, below, gbx-lm runs it on the int8 tensor cores.

On an NVIDIA DGX Spark

Measured 2026-10-22 on an NVIDIA DGX Spark (GB10, 128 GB), both runtimes on the same machine with the same prompts and settings as above.

Decode speed in tokens per second; in brackets, how many times as fast gbx-lm with this draft head is. Versions: gbx-lm 0.8.2; llama.cpp Prism fork 73bfe6c (CUDA, flash attention on) with the PQ2_0 GGUF, the same 2-bit packing of the ternary weights as the MLX pack. Without the draft head gbx-lm still decodes 12% faster than llama.cpp.

Prompt processing: 1.6× as fast as llama.cpp

Tokens per second; in brackets, how many times as fast gbx-lm is. The ternary products run on the GB10's int8 tensor cores, with activations in int8 blocks of 128 (GBX_TERNARY_INT8=off keeps them exact).

Long conversations are faster in v0.8.3: with a 54k-token document already in the conversation, the first answer takes 42 s (92 s in v0.8.2) and the next 1.3k-token message 1.6 s (4.5 s). And a new prompt length no longer costs a one-time pause of about 3 s.

Memory

On the Spark gbx-lm keeps only the 1.6-bit form of the weights in memory: 5.9 GB for the model after loading (llama.cpp's PQ2_0 file is 7.2 GB).

Requirements

Mac

Linux

The model and its draft head take about 10 GB of memory, plus the conversation's cache.

Install

# upgrading? clear the previous version's unpack directory first
rm -rf ~/.libra/cache/onefile/gbx_lm

# on a DGX Spark, use gbx_lm-linux-arm64.tar.gz in place of gbx_lm-darwin-arm64.tar.gz
curl -fL -o gbx_lm-darwin-arm64.tar.gz 'https://github.com/GreenBitAI/gbx-lm/releases/latest/download/gbx_lm-darwin-arm64.tar.gz' \
  && tar -xzf gbx_lm-darwin-arm64.tar.gz gbx_lm \
  && mkdir -p "$HOME/.local/bin" \
  && mv gbx_lm "$HOME/.local/bin/gbx_lm" \
  && chmod +x "$HOME/.local/bin/gbx_lm"

gbx_lm -h

command not found -- $HOME/.local/bin is not on your PATH:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc     # zsh
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bash_profile && source ~/.bash_profile   # bash

Killed: 9 (Mac) -- a previous version's files are still in the unpack directory, and macOS refuses to mix two builds. Run the rm -rf line above, then try again.

Run

GBX_QWEN35_MTP=on gbx_lm --model GreenBitAI/Ternary-Bonsai-2-27B-gbx

That serves an OpenAI-compatible API on port 11688, its default. The weights download on first use into ~/.libra/cache/models; set HF_HOME to put them elsewhere. Leave out GBX_QWEN35_MTP=on to run without the draft head.

curl http://127.0.0.1:11688/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"GreenBitAI/Ternary-Bonsai-2-27B-gbx","messages":[{"role":"user","content":"Hello"}]}'

The model reads images as well as text, through the vision tower Prism ML ships with it. Send them as OpenAI image_url parts -- an http(s) URL or a base64 data: URL -- or pass --images photo.jpg to gbx_lm generate.

From the command line

GBX_QWEN35_MTP=on gbx_lm generate --model GreenBitAI/Ternary-Bonsai-2-27B-gbx --prompt "Hello" --max-tokens 2048
GBX_QWEN35_MTP=on gbx_lm chat --model GreenBitAI/Ternary-Bonsai-2-27B-gbx --max-tokens 2048

Keep --max-tokens generous: the model reasons before it answers, and that counts against it.

Coding agents

The server speaks three wire protocols on the same port:

Codex -- a provider in ~/.codex/config.toml:

[model_providers.gbx]
name = "gbx-lm"
base_url = "http://127.0.0.1:11688/v1"
wire_api = "responses"

and a profile in ~/.codex/gbx.config.toml:

model_provider = "gbx"
model = "GreenBitAI/Ternary-Bonsai-2-27B-gbx"
model_context_window = 262144

Claude Code -- ~/.claude/gbx.settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://127.0.0.1:11688",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "GreenBitAI/Ternary-Bonsai-2-27B-gbx",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "GreenBitAI/Ternary-Bonsai-2-27B-gbx"
  }
}

Both clients ask for a small model for their own background work, so every name in the settings has to be one this server is serving.

What is in here

• Everything outside mtp/ is prism-ml/Ternary-Bonsai-2-27B-mlx-2bit, unchanged: Apache 2.0, copyright Prism ML. Created using Bonsai by Prism ML.
• mtp/mtp.safetensors is the draft head, trained by GreenBit AI from the multi-token prediction head of Qwen/Qwen3.8-27B and distilled on Bonsai 2's own output. Apache 2.0.

See NOTICE.txt.