Qwen3-32B-lecore-19g-mlx

MLX mixed-precision quantization of Qwen/Qwen3-32B, with the bit width of every linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.

How the bits were chosen

1. Each decoder linear tensor was quantized alone (MLX affine, group 64 during measurement) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
2. Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
3. Converted with mlx_lm.convert and a per-layer quant_predicate. Bit histogram (linear tensors): {'2': 0, '3': 0, '4': 448, '8': 0, '16': 0} Note: at this size budget the allocator kept every linear tensor at the top rung (4 bits), so the per-tensor selection made no downgrades. In practice this export is uniform 4-bit (group 64), with 4-bit embeddings and a 6-bit lm_head. It is not a mixed-precision mix, and the fair baseline is stock MLX 4-bit, not 3-bit Note: at this size budget the allocator kept every linear tensor at the top rung (4 bits), so the per-tensor selection made no downgrades. In practice this export is uniform 4-bit (group 64), with 4-bit embeddings and a 6-bit lm_head. It is not a mixed-precision mix, and the fair baseline is stock MLX 4-bit, not 3-bit.

Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)

Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU). "bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.

Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)

MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.

Only measured numbers are reported. Downstream benchmarks are limited to the MMLU/GSM8K subsets above. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.

GSM8K regression (diagnosed)

GSM8K is much lower than both bf16 and stock 3-bit, even though perplexity is close to bf16. We ran a diagnostic on the first 20 GSM8K problems:

The quantized model reasons correctly but in a long "thinking" style, so it runs past the 320-token limit before writing the #### answer line. The drift comes from the 4-bit MLP weights. If you need few-shot math in this format, use stock quantizations or a larger budget. For chat use with a longer generation limit this matters less, but we have not measured that.

Files in this repo

Usage (Apple Silicon)

pip install -U mlx-lm
mlx_lm.generate --model staccs/Qwen3-32B-lecore-19g-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192

Weights are ~18.62 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.

License

Base model: Qwen/Qwen3-32B, released by the Qwen team under the Apache License 2.0 (https://huggingface.co/Qwen/Qwen3-32B/blob/main/LICENSE). Only quantization was done here: no retraining.

Related builds

Same base model, other formats in the lecore set: