Qwen3-32B-lecore-mlx
MLX mixed-precision quantization of Qwen/Qwen3-32B, with the bit width of every
linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.
How the bits were chosen
- Each decoder linear tensor was quantized alone (MLX affine, group 64) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
- Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
- Converted with
mlx_lm.convertand a per-layerquant_predicate. Bit histogram (linear tensors): {'2': 177, '3': 190, '4': 81, '8': 0}.
Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)
Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU).
"bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.
| Model | Size (GB, weights) | bpw | WikiText-2 ppl (32704 tok) | Python-code ppl (8176 tok) |
|---|---|---|---|---|
| bf16 reference | – | 16 | 9.7287 | 3.5031 |
| this repo (leCore Unicron mix) | 13.00 | 3.173 | 13.4162 | 4.5586 |
| stock MLX 3-bit, group 64 | 14.33 | 3.500 | 39.0780 | 7.4644 |
Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)
MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.
| Model | MMLU (5-shot) | GSM8K strict | GSM8K flexible |
|---|---|---|---|
| leCore mix (this repo) | 80.0% | – | – |
Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.
Files in this repo
| file | size | what it is |
|---|---|---|
model-00001-of-00003.safetensors |
5.35 GB | quantized weights |
model-00002-of-00003.safetensors |
5.36 GB | quantized weights |
model-00003-of-00003.safetensors |
2.29 GB | quantized weights |
tokenizer.json |
11.4 MB | tokenizer |
vocab.json |
2.8 MB | config / tokenizer |
merges.txt |
1.7 MB | support file |
model.safetensors.index.json |
137.9 KB | config / tokenizer |
config.json |
97.5 KB | model config |
tokenizer_config.json |
5.4 KB | config / tokenizer |
chat_template.jinja |
4.2 KB | chat template |
added_tokens.json |
707 B | config / tokenizer |
Usage (Apple Silicon)
pip install -U mlx-lm
mlx_lm.generate --model staccs/Qwen3-32B-lecore-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192
Weights are ~13.00 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.
License
Base model: Qwen/Qwen3-32B, released by the Qwen team under the Apache License 2.0 (https://huggingface.co/Qwen/Qwen3-32B/blob/main/LICENSE). Only quantization was done here: no retraining.
Related builds
Same base model, other formats in the lecore set:
| repo | format | size |
|---|---|---|
| Qwen3-32B-lecore-19g-GGUF | GGUF (llama.cpp) | 38.40 GB |
| Qwen3-32B-lecore-19g-mlx | MLX (Apple Silicon) | 18.64 GB |
| Qwen3-32B-lecore-GGUF | GGUF (llama.cpp) | 32.88 GB |
| Qwen3-32B-lecore-hrr | gated-HRR gates | 4.4 MB |