Prism ML Website  |  Demo & Examples  |  Discord

Bonsai 2 Ternary 4B: MLX

A 4B-class vision-language model with ternary transformer weights, packed for MLX on Apple Silicon.

1.87 GB on disk, language model + vision tower (FP16 language model: ~8.4 GB)

Highlights

• Ternary language weights stored in MLX's 2-bit grouped format (group 128). MLX stores a scale and a bias per group, so the container costs a little more than the GGUF packings; the weights decode to exactly the same ternary values.
• Vision input through the official vision tower, carried unrotated in FP16.
• 262K-token context, inherited from the Qwen3.5-4B hybrid-attention backbone (24 linear-attention and 8 full-attention layers).

Model Overview

Files

Quickstart

PrismML-Eng/Bonsai-demo is the source of truth for running these models. Where anything here disagrees with it, it is right.

hf download prism-ml/Ternary-Bonsai-2-4B-mlx-2bit --local-dir bonsai2-4b-mlx
pip install "mlx-vlm>=0.7.8"

mlx-vlm 0.7.8 and newer load this pack natively (model_type: prism_hadamard_qwen35):

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("bonsai2-4b-mlx")
prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=4096,
               temperature=1.0, top_p=0.95, top_k=20))

For an OpenAI-compatible server, run python -m mlx_vlm.server --model bonsai2-4b-mlx and send the sampling settings in each request.

The released mlx-lm (0.32.0) cannot load this pack yet; its support is merged and arrives with the next mlx-lm release. On older mlx-vlm versions, use the loader bundled in runtime/ (pip install -r bonsai2-4b-mlx/runtime/requirements.txt, then python bonsai2-4b-mlx/quickstart.py "your prompt"); PACK-RUNTIME.md documents it. quickstart.py is a smoke test: it answers with thinking off and stops at 256 tokens.

Pass no images for text-only use.

For CUDA, CPU, and llama.cpp on Metal, use the GGUF packs of the same weights: Ternary-Bonsai-2-4B-gguf.

Best Practices

Generation Parameters

We recommend these sampling settings (the same as the GGUF packs):

• Thinking mode (default): temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.0, repetition_penalty=1.0
• Non-thinking mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Pass the sampler explicitly. mlx-vlm's Python generate() decodes greedily unless you pass these arguments. mlx_vlm.server takes temperature, top_p and top_k from generation_config.json when a request omits them, but not presence penalty or min_p, so send the full set per request. Apply penalties over the last 64 tokens where your client supports a penalty window. Avoid greedy decoding with thinking on; it can get stuck reasoning on some prompts. To turn thinking off, render the chat template with enable_thinking=False.

Output Length

Give thinking mode room to finish: even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.

System Prompt

You can use a simple system prompt such as:

You are a helpful assistant

Speculative Decoding (optional)

mtp/ holds this model's trained multi-token-prediction head as an mlx-vlm drafter. The main model verifies every drafted token, so the output matches plain decoding (up to float16 near-ties). It helps most on copy-heavy work such as code edits and rewrites; on general chat it is about as fast as plain decoding or slightly slower, so plain decoding stays the default. MTP.md has the code. It was validated with the bundled runtime on mlx-vlm 0.6.3 and has not yet been re-checked on 0.7.8.

License

Apache 2.0, following the base model.