Nanbeige4.3-33B MLX, mixed 3/4/6-bit (15.7 GiB)

MLX quantization of Nanbeige/Nanbeige4.3-33B (dense 33B Llama, 52 layers) sized to run on a 24 GB Apple Silicon Mac.

The quality-first variant: 3-bit only where the model is most tolerant (MLP gate/up), 4-bit elsewhere, 6-bit lm_head. Quantized directly from the original BF16 weights.

Quantization layout (affine, group size 64)

Measured on Apple M5 Pro, 24 GB

• Load 3–5 s, 15.7 GiB active, ~17 GB peak during generation
• Decode ~15–16.5 tok/s (bandwidth bound: every token reads all weights)
• 7/8 LeetCode-style problems pass hidden tests (Two Sum, Merge Intervals, Longest Substring, LRU Cache, Word Ladder, Trapping Rain Water, Median of Two Sorted Arrays; fails Regex Matching without reasoning)
• As a sandboxed terminal agent (native XML tool calls, bash tool), it found and fixed 3 bugs in a small repo and verified with the test suite in 8 commands, about a minute

Usage

pip install -U mlx-lm
mlx_lm.generate --model alexwengg/Nanbeige4.3-33B-MLX-mixed-3-4-6bit \
  --prompt "Which number is bigger, 9.11 or 9.8?" --max-tokens 512 --temp 0.6 --top-p 0.95

from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("alexwengg/Nanbeige4.3-33B-MLX-mixed-3-4-6bit")
msgs = [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}]
# enable_thinking=True (default) streams reasoning first; False answers directly.
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False)
for r in stream_generate(model, tok, prompt, max_tokens=1024, sampler=make_sampler(temp=0.6, top_p=0.95)):
    print(r.text, end="", flush=True)

OpenAI-compatible server: mlx_lm.server --model alexwengg/Nanbeige4.3-33B-MLX-mixed-3-4-6bit.

Faster with speculative decoding

Nanbeige/Nanbeige4.1-3B uses exactly the same tokenizer, so it works as a draft model:

mlx_lm.convert --hf-path Nanbeige/Nanbeige4.1-3B --mlx-path nanbeige-4.1-3b-3bit -q --q-bits 3
mlx_lm.generate --model alexwengg/Nanbeige4.3-33B-MLX-mixed-3-4-6bit --draft-model nanbeige-4.1-3b-3bit --num-draft-tokens 3 \
  --prompt "Write a Python LRU cache class." --max-tokens 512

With this repo plus a 4-bit draft, code went from 15.3 to 25–27 tok/s (63–76% of draft tokens accepted) and prose from 15.2 to ~19 tok/s. Together they peak around 19.3 GB, which is tight on a 24 GB Mac: close other apps first, or use the 3-bit repo for more headroom.

Model card recommendations: temperature 0.6 / top-p 0.95 for chat and reasoning, 1.0 / 0.95 for agentic use. The chat template supports enable_thinking, preserve_thinking, and tool calls (tool_call_format="xml" recommended).

Notes

• Speeds are single-stream decode with nothing else heavy running; under memory pressure (browser + chat apps on a 24 GB machine) expect swapping and lower numbers.
• Pass rates are from a small 8-problem set at temperature 0.6 without reasoning, so treat them as a smoke test, not a benchmark.
• Converted shard by shard (download → quantize → delete), so the 66 GB BF16 checkpoint never had to fit on disk or in memory.

License: Apache-2.0, same as the original model. All credit for the model goes to the Nanbeige LLM Lab.