Nanbeige4.3-33B MLX, 3-bit (13.9 GiB)

MLX quantization of Nanbeige/Nanbeige4.3-33B (dense 33B Llama, 52 layers) sized to run on a 24 GB Apple Silicon Mac.

The speed and headroom variant: 3-bit everywhere except a 6-bit lm_head. Leaves room on a 24 GB Mac for a speculative draft model.

Quantization layout (affine, group size 64)

Measured on Apple M5 Pro, 24 GB

• Load ~4 s, 13.9 GiB active
• Decode ~17.5 tok/s alone; with the 3-bit Nanbeige4.1-3B draft, 25–29 tok/s on code (~70% of draft tokens accepted) and ~21 tok/s on chat, ~15.5 GiB total
• 5/8 LeetCode-style problems pass hidden tests (vs 7/8 for the mixed variant)
• Handles short agent tasks well (sandboxed bash tool: tidy a folder in ~30 s, find largest files in ~15 s)

Usage

pip install -U mlx-lm
mlx_lm.generate --model alexwengg/Nanbeige4.3-33B-MLX-3bit \
  --prompt "Which number is bigger, 9.11 or 9.8?" --max-tokens 512 --temp 0.6 --top-p 0.95

from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("alexwengg/Nanbeige4.3-33B-MLX-3bit")
msgs = [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}]
# enable_thinking=True (default) streams reasoning first; False answers directly.
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False)
for r in stream_generate(model, tok, prompt, max_tokens=1024, sampler=make_sampler(temp=0.6, top_p=0.95)):
    print(r.text, end="", flush=True)

OpenAI-compatible server: mlx_lm.server --model alexwengg/Nanbeige4.3-33B-MLX-3bit.

Faster with speculative decoding

Nanbeige/Nanbeige4.1-3B uses exactly the same tokenizer, so it works as a draft model:

mlx_lm.convert --hf-path Nanbeige/Nanbeige4.1-3B --mlx-path nanbeige-4.1-3b-3bit -q --q-bits 3
mlx_lm.generate --model alexwengg/Nanbeige4.3-33B-MLX-3bit --draft-model nanbeige-4.1-3b-3bit --num-draft-tokens 3 \
  --prompt "Write a Python LRU cache class." --max-tokens 512

This is the variant to use with a draft on a 24 GB Mac: 13.9 GiB + 1.6 GiB draft.

Model card recommendations: temperature 0.6 / top-p 0.95 for chat and reasoning, 1.0 / 0.95 for agentic use. The chat template supports enable_thinking, preserve_thinking, and tool calls (tool_call_format="xml" recommended).

Notes

• Speeds are single-stream decode with nothing else heavy running; under memory pressure (browser + chat apps on a 24 GB machine) expect swapping and lower numbers.
• Pass rates are from a small 8-problem set at temperature 0.6 without reasoning, so treat them as a smoke test, not a benchmark.
• Quality caveat: the 4-bit layers of the mixed 3/4/6-bit repo were re-quantized to 3-bit (dequantize → quantize), not quantized from BF16 directly, so it carries slightly more error than a direct 3-bit conversion. If you have the memory, prefer the mixed variant.

License: Apache-2.0, same as the original model. All credit for the model goes to the Nanbeige LLM Lab.