Ternary Bonsai 2 27B — CMF Q2TP

CMF conversion of Prism ML's Ternary Bonsai 2 27B MLX pack, supported by Cortiq 0.6.9.

What is stored

The 402 large packed matrices use existing CMF dtype16 Q2TP, with explicit required affine and signed-Hadamard operators. The affine operator preserves ternary signs and zeros; CMF's fitted scale ladder introduces a small scale approximation. This is not a Q4TP conversion, a new Q2TP-T dtype, or a claim of byte-identical upstream weights.

Auxiliary and normalization tensors retain their declared precisions. Older runtimes must reject unsupported required features; stock 0.6.8 does not support this model profile. The 0.6.9 runtime includes resident Vulkan inference and native Metal Q2 kernels; its Metal ternary optimization operates directly on the packed two-bit weights.

Install

Use the matching platform asset from v0.6.9 releases, or install from crates.io:

cargo install cortiq-cli --version 0.6.9 --locked --features gpu
cortiq --version

Linux needs a working Vulkan driver; confirm the GPU appears in vulkaninfo --summary. On macOS, use an Apple Silicon build for native Metal. For source builds, see SOURCE_BUILD.md. No custom patch is needed for 0.6.9.

Download and verify

curl -L --fail -o Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  https://huggingface.co/infosave/Ternary-Bonsai-2-27B-cmf/resolve/main/Ternary-Bonsai-2-27B-Q2TP-Affine.cmf
# Linux:
sha256sum Ternary-Bonsai-2-27B-Q2TP-Affine.cmf
# macOS: shasum -a 256 Ternary-Bonsai-2-27B-Q2TP-Affine.cmf

cortiq verify Ternary-Bonsai-2-27B-Q2TP-Affine.cmf

Compare the complete hash with the table above.

Run: Linux / Vulkan

export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/tmp}"
export WGPU_BACKEND=vulkan CMF_GPU=wgpu
export CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_GPU_WGPU_GRAPH=1 CMF_MTP=0

cortiq run Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  --prompt 'Return only a JSON object with city equal to Казань and count equal to 3.' \
  --greedy --seed 42 --no-think --max-tokens 128

Run: Linux / CUDA

Cortiq 0.8.96 or newer, built with the CUDA backend (the NVIDIA driver and the CUDA 12.x libraries):

cargo install cortiq-cli --features cuda
CMF_GPU=cuda cortiq run Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  --prompt 'Return only a JSON object with city equal to Казань and count equal to 3.' \
  --greedy --seed 42 --no-think --max-tokens 128

On an RTX A5000 (24 GB) the whole token runs on the card: cortiq bench decodes at 33.8–33.9 tok/s and reads a 4096-token prompt at 777 tok/s (first token 5.3 s); Vulkan on the same card gives 25.8 tok/s and 122 tok/s (33.7 s).

Run: Apple Silicon / native Metal

In a separate shell, without the Vulkan variables:

export CMF_GPU=1 CMF_Q2TP_GPU=1 CMF_GPU_PROBE=0 CMF_MTP=0
export CMF_METAL_PREFILL_CHUNK=32

cortiq run Ternary-Bonsai-2-27B-Q2TP-Affine.cmf \
  --prompt 'Explain in two sentences why the sky looks blue.' \
  --greedy --seed 42 --no-think --max-tokens 128

Chunked Metal prefill processes prompt rows in batches; generation uses the native ternary kernel. The one-command-buffer experiment was not faster and is not required. The failed experimental Q8/Q8×2 and alternative packing kernels are not recommended runtime settings.

Measured results and limitations

These are engineering measurements of the accepted development implementations, not universal hardware guarantees or a claim that every release build was benchmarked.

The matched throughput test used 1,122 identical prompt IDs and 128 greedy output tokens, one resident model, one cold run and three warm runs. Decode timing covers 127 callback intervals, separately from prefill and model loading. The upstream reference uses its own internal timing boundary; small boundary differences remain.

20 tok/s on this M4 was not achieved. Prompt processing and time-to-first-token remain separate costs; longer conversations, temperature/clock conditions and application-side sampling affect throughput. The 8.25 GB file size is not peak VRAM/RAM usage: activations, prefill scratch and KV/recurrent state require additional memory.

Quality correspondence

Frozen mixed English/Russian/code corpus: 1,122 token IDs, two independent reset slices [0,513) and [512,1025), scoring 512 next-token targets per slice (1,024 total), without a synthetic BOS.

This small correspondence corpus is not GPQA, Terminal-Bench, AA-LCR, MMMU-Pro or a broad model-quality evaluation. Six deterministic text checks also covered arithmetic, structured JSON, factual context, English/Russian explanations and a Python function. The fixed throughput test retained identical output IDs after the accepted Metal change; that does not imply identical outputs for every prompt.

The separate ordinary, lossy CMF comparison was a mixed Q2TP/Q4TP candidate and failed the +0.05-NLL acceptance threshold. It is not the model distributed here. Plain Q8 activation experiments also failed the component-fidelity gate. A two-component CUDA activation experiment preserved fidelity but was not integrated as a production CUDA backend; no slice timing is presented as model throughput.

Scope

• Text-only runtime support. Preserved vision records do not imply working multimodal inference.
• The measurements above use ordinary attention, not an O(1)-attention quality certification for Bonsai.
• No promise of unlimited context, fixed total runtime memory, CUDA parity, broad benchmark scores or lossless source equivalence.
• Use 0.6.9, not the historical 0.6.8-labelled patched development binary.

Provenance and licensing

Upstream revision: 3f926b415992eaa2ae9dd7b573706494d6bbf787. Original safetensors SHA-256: 130de5925082c168b7866b2e91b52e44abbafc99017e3ca352b77b5b55a269ed.

See runtime provenance, the frozen inputs under evaluation/, and the retained license bundle. Preserve the upstream model LICENSE/NOTICE, MLX runtime notice and CMF notices when redistributing. Historical source patches remain for reproducibility of old measurements; they are not the installation method for 0.6.9.