QwQ-32B-lecore-mlx
MLX mixed-precision quantization of Qwen/QwQ-32B, with the bit width of every
linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.
How the bits were chosen
- Each decoder linear tensor was quantized alone (MLX affine, group 64 during measurement) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
- Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
- Converted with
mlx_lm.convertand a per-layerquant_predicate. Bit histogram (linear tensors): {'2': 157, '3': 218, '4': 73, '8': 0}.
Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)
Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU).
"bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.
| Model | Size (GB, weights) | bpw | WikiText-2 ppl (32704 tok) | Python-code ppl (8176 tok) |
|---|---|---|---|---|
| bf16 reference | – | 16 | 7.6332 | 3.4185 |
| this repo (leCore Unicron mix) | 12.99 | 3.172 | 12.9810 | 4.7021 |
| stock MLX 3-bit, group 64 | 14.34 | 3.500 | 11.3413 | 4.3705 |
Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)
MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.
| Model | MMLU (5-shot) | GSM8K strict | GSM8K flexible |
|---|---|---|---|
| leCore mix (this repo) | 77.2% | – | – |
Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.
Files in this repo
| file | size | what it is |
|---|---|---|
model-00001-of-00003.safetensors |
5.37 GB | quantized weights |
model-00002-of-00003.safetensors |
5.34 GB | quantized weights |
model-00003-of-00003.safetensors |
2.28 GB | quantized weights |
tokenizer.json |
11.4 MB | tokenizer |
vocab.json |
2.8 MB | config / tokenizer |
merges.txt |
1.7 MB | support file |
model.safetensors.index.json |
143.1 KB | config / tokenizer |
config.json |
97.4 KB | model config |
tokenizer_config.json |
5.4 KB | config / tokenizer |
chat_template.jinja |
2.8 KB | chat template |
added_tokens.json |
707 B | config / tokenizer |
Usage (Apple Silicon)
pip install -U mlx-lm
mlx_lm.generate --model staccs/QwQ-32B-lecore-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192
Weights are ~12.99 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.
License
Base model: Qwen/QwQ-32B, released by the Qwen team under the Apache License 2.0 (https://huggingface.co/Qwen/QwQ-32B/blob/main/LICENSE). Only quantization was done here: no retraining.
Related builds
Same base model, other formats in the lecore set:
| repo | format | size |
|---|---|---|
| QwQ-32B-lecore-GGUF | GGUF (llama.cpp) | 32.96 GB |
| QwQ-32B-lecore-hrr | gated-HRR gates | 2.8 MB |