Nanbeige4.3-33B · MLX 4-bit (AWQ)
Nanbeige/Nanbeige4.3-33B (revision c1093c50) for Apple silicon:
every decoder layer in activation-aware 4-bit (AWQ, MLX affine, group 64), the embedding and the LM head in 8-bit.
19.8 GB, 4.79 bits per weight. Runs in mlx-lm and in
Kapitsa.
| Base model | Nanbeige4.3-33B: dense Llama, 52 layers, 56 query / 8 KV heads, 33.1B parameters, 512K context |
| Format | MLX safetensors, affine quantization, group size 64, bf16 scales and biases |
| Layers | AWQ 4-bit (mlx_lm.awq, 128 calibration samples) |
| Embedding, LM head | 8-bit |
| Size | 19.8 GB on disk, 4.79 bits per weight |
| Memory | a 32 GB Mac with a short context; 48 GB for 64K tokens; 64 GB for 128K (the KV cache is 208 KB a token in fp16) |
| Other packs | 8-bit (35.2 GB) · bf16 (66.3 GB) |
Quality
Every pack below scored on the same test against the bf16 model: 768 next-token positions in 8 windows of Python
source (512-3,224 tokens, each starting with <|im_start|> as a chat does), computed by Kapitsa on an M5 Max. The
GGUFs were made with llama.cpp b11871 from the same checkpoint and dequantized back to bf16 so they ran through the
same engine; their numbers are the GGUF weights', not llama.cpp's arithmetic.
| Pack | Size | Bits/weight | KL mean | KL median | KL p99 | Same top token |
|---|---|---|---|---|---|---|
| MLX 8-bit (ours) | 35.2 GB | 8.50 | 0.0155 | 0.00003 | 0.18 | 98.6% |
| llama.cpp Q8_0 | 35.2 GB | 8.50 | 0.0059 | 0.00001 | 0.12 | 99.0% |
| llama.cpp Q4_K_M | 20.1 GB | 4.84 | 0.144 | 0.0010 | 2.65 | 93.5% |
| this pack: AWQ 4-bit, 8-bit embedding and head | 19.8 GB | 4.79 | 0.163 | 0.0015 | 3.77 | 92.3% |
AWQ 4-bit, 4-bit embedding and head (this repo at 08c5b29c) |
18.6 GB | 4.50 | 0.181 | 0.0021 | 3.69 | 92.1% |
plain 4-bit (mlx_lm.convert -q) |
18.6 GB | 4.50 | 0.267 | 0.0030 | 5.27 | 90.5% |
AWQ removes a third of plain 4-bit's divergence at the same size, and 8 bits for the embedding and LM head another tenth for 1.2 GB more. llama.cpp's Q4_K_M still scores better on this test, at a similar size: its K-quant blocks carry finer scales, and it keeps half the layers' value and down projections (26 of 52) and its LM head at 6 bits. The mean KL is carried by a few hard positions; the median position is close to bf16 in every 4-bit pack. KL is measured in nats per position.
Speed
MacBook Pro M5 Max (128 GB, macOS 26.4). Prefill of a prompt, then decode at that context (tokens/s). Kapitsa and mlx-lm 0.32.2 ran this pack; llama.cpp b11871 ran a Q4_K_M of the same checkpoint. Each engine ran in turn at each length, with the GPU quiet before each run.
| Context | prefill: Kapitsa / mlx-lm / llama.cpp | decode: Kapitsa / mlx-lm / llama.cpp |
|---|---|---|
| 1K | 730 / 823 / 665 | 28.2 / 28.1 / 26.7 |
| 4K | 678 / 773 / 555 | 27.0 / 27.2 / 25.0 |
| 16K | 511 / 501 / 345 | 23.3 / 21.8 / 17.9 |
| 32K | 387 / 455 / 295 | 20.0 / 18.6 / 16.2 |
| 64K | 324 / 410 / 233 | 16.0 / 15.4 / 12.6 |
| 128K | 242 / 277 / 140 | 11.3 / 11.0 / 8.2 |
Runs swing by up to about 20% as the chip heats.
On real chat prompts (Kapitsa's benchmark harness: Nanbeige's chat template, 5K- and 20K-token prompts, T 0.6, thinking off), every MLX engine on this pack and the GGUF engines on Q4_K_M:
| Engine | Decode (tok/s) | Prefill 4K / 16K (tok/s) | Peak memory |
|---|---|---|---|
| Kapitsa | 28.8 | 532 / 483 | 25.3 GB |
| oMLX | 28.0 | 766 / 634 | 35.2 GB |
| mlx-vlm server | 27.6 | 705 / 589 | 34.6 GB |
| mlx-lm server | 27.0 | 668 / 573 | 40.6 GB |
| llama.cpp server | 25.9 | 496 / 394 | 33.3 GB |
| Ollama | 25.6 | 503 / 394 | 14.5 GB + 18.4 GB mmap |
The runs, logs and scripts are in Kapitsa's repository.
Run
Kapitsa 0.2.18 or later (Apple silicon, Metal):
brew install nace-ai/tap/kapitsa # or: curl -fsSL https://raw.githubusercontent.com/nace-ai/kapitsa/main/install.sh | bash
kapitsa serve --model nanbeige # downloads this pack, serves an OpenAI-compatible API on localhost
kapitsa serve --model nanbeige:8bit # the 8-bit pack
mlx-lm:
pip install mlx-lm
mlx_lm.generate --model kapitsaprime/Nanbeige4.3-33B-MLX-4bit --prompt "Write a Python function that merges two sorted lists."
mlx_lm.server --model kapitsaprime/Nanbeige4.3-33B-MLX-4bit
The chat template, tokenizer and generation settings are the original model's. Reasoning goes in <think> and is on
by default; enable_thinking: false in the template arguments turns it off. Tool calls are XML in <tool_call>.
Nanbeige recommends temperature 0.6 and top-p 0.95 for reasoning and chat, 1.0 and 0.95 for agentic use.
How it was made
mlx_lm.awq --model Nanbeige/Nanbeige4.3-33B --mlx-path awq --bits 4 --group-size 64 --embed-bits 4 --embed-group-size 64
then the embedding and LM head re-quantized from the bf16 checkpoint at 8 bits, group 64 (mx.quantize; AWQ leaves
both unscaled, so they are the plain checkpoint's tensors). mlx-lm 0.32.2, mlx 0.32.4, 77 minutes on an M5 Max;
mlx_lm.awq's defaults otherwise: 128 samples of 512 tokens from its calibration text, a 20-point scale grid, seed 123.
License and credits
Apache 2.0, as the base model. Weights by Nanbeige; quantized and measured by Nace AI with Kapitsa.