Bonsai-27B — CMF q1 (4.75 GB, runs on a 24 GB MacBook)

Bonsai-27B — a 1-bit-trained 27B hybrid (16 full-attention + 48 GatedDeltaNet layers, 248K vocabulary) — packed into a single CMF file with q1 encoding: 1.5 bits per weight, 6 bytes per 32 weights.

Because the model was trained at 1 bit, q1 is its native representation, not a lossy afterthought: generation is token-for-token identical to the same model stored at q8. The whole model is one 4.75 GB file that memory-maps straight off disk and decodes at 9–11 tok/s on an Apple M4 MacBook (24 GB) through a whole-token Metal graph and at 61 tok/s on an RTX A5000 with CUDA.

What is CMF?

CMF (Cortiq Model Format) is a single-file LLM container with a small pure-Rust runtime — no Python, no torch, no C++ toolchain, no CUDA install:

• One file carries the weights, tokenizer and chat template, and checks its own integrity.
• mmap-first: pages load lazily on first touch; start-up is fast and RAM stays near the file size.
• Per-tensor quantization (q8 / q4 / vbit / q1 for 1-bit-trained models).
• O(1) attention option (--o1): convert attention to a constant-memory streaming operator — no retraining, weights byte-identical.
• Skills: one file can carry a swarm of specialists sharing a base model, with self-routing. See the skills guide.

Format spec and engine: https://github.com/infosave2007/cmf Mobile app (Flutter, runs CMF on-device): https://github.com/infosave2007/cmfmobile — note this 4.75 GB model needs a 16–24 GB machine; on phones use Bonsai-1.7B in the same format (334 MB).

Install

Prebuilt binaries (macOS arm64/x86_64, Linux, Windows): https://github.com/infosave2007/cmf/releases/latest

Or with a Rust toolchain:

cargo install cortiq-cli   # needs >= 0.3.6

Download & run

# download the model file (~4.75 GB)
huggingface-cli download infosave/Bonsai-27Bcmf bonsai-27b-q1.cmf --local-dir .

# chat (the file carries its own chat template)
cortiq run bonsai-27b-q1.cmf --prompt "Explain mmap in one sentence."

# Apple Silicon: run the whole-token Metal graph (9-11 tok/s on an M4)
CMF_GPU=1 cortiq run bonsai-27b-q1.cmf --prompt "Explain mmap in one sentence."

# NVIDIA: the CUDA backend (a build with `cargo install cortiq-cli --features cuda`)
CMF_GPU=cuda cortiq run bonsai-27b-q1.cmf --prompt "Explain mmap in one sentence."

# raw completion mode (no chat template)
cortiq run bonsai-27b-q1.cmf --prompt "The capital of France is" --raw --greedy

Run as a server

cortiq serve speaks the OpenAI API, so existing clients and SDKs work unchanged:

CMF_GPU=1 cortiq serve bonsai-27b-q1.cmf --port 8080   # + web dashboard on /

curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "cmf",
  "messages": [{"role": "user", "content": "Explain mmap in one sentence."}]
}'

Streaming ("stream": true), /v1/models, /v1/completions and /healthz work too. Note the scope honestly: requests are serialized (one at a time per model) and there is no authentication — a local-first server, not a multi-tenant gateway.

Performance (Apple M4, 24 GB MacBook)

The GPU decode is distribution-equivalent to the CPU path (reduction order differs — the usual GPU tolerance class).

Performance (RTX A5000 24 GB, CUDA, cortiq 0.8.96)

cortiq bench --ignore-eos (64 tokens), CMF_GPU=cuda on a build with the CUDA backend (the NVIDIA driver and the CUDA 12.x libraries):

The greedy text is identical to the CPU's. Vulkan on the same card: decode 26.3–26.9 tok/s, the 4096-token prompt 32 tok/s (first token 128 s).

Provenance

Converted from prism-ml/Bonsai-27B-unpacked (Apache-2.0) with:

cortiq convert --model prism-ml/Bonsai-27B-unpacked --quant q1 --output bonsai-27b-q1.cmf

q1 is intended for 1-bit-trained checkpoints (Bonsai / BitNet class). As post-training quantization of an ordinary model it destroys quality — the converter exposes it only as an explicit opt-in.