4-bit post-training quantization of Qwen/Qwen3-0.6B for Apple Silicon, produced with GPTQ using a float32 Hessian.
This model is an artifact of the research project Edge AI & Quantization, a pre-registered comparative study of post-training quantization (RTN, GPTQ, AWQ, DWQ and K-quants) of small LLMs (0.6B to 4B) on a real edge device.
In mlx-lm 0.32.2, the stock GPTQ accumulates the Hessian (XᵀX) in bfloat16. On the 3B to 4B models of this study, rounding makes the Hessian of some late layers lose positive semi-definiteness, and the quantized model returns NaN perplexity. Accumulating the Hessian in float32 (gptq_camada.py --hessiana fp32) removes the failure: all 10 jobs (4 and 3 bits, 5 models) completed with 0 NaN. The diagnosis is documented in LOGICA-QUANTIZACAO.md.
Measured on a MacBook Pro (Apple M5 Pro, 24 GB unified memory), mlx-lm 0.32.2, mlx 0.32.4.
| fp16 (original) | This model | |
|---|---|---|
| Size on disk | 1.20 GB | 0.39 GB (3.1× smaller) |
| Perplexity, WikiText-2 (lower is better) | 20.97 | 22.58 |
| Mean zero-shot accuracy, 5 tasks | 0.5215 | 0.4958 |
| Perplexity retention Q = PPL(fp16) / PPL(quantized) | 1.000 | 0.9285 |
pip install mlx-lm
mlx_lm.generate --model andre-atlas/Qwen3-0.6B-GPTQ-fp32-4bit-mlx --prompt "Explain quantization in one paragraph."
from mlx_lm import load, generate
model, tokenizer = load("andre-atlas/Qwen3-0.6B-GPTQ-fp32-4bit-mlx")
print(generate(model, tokenizer, prompt="Explain quantization in one paragraph.", max_tokens=200))
Apache-2.0, the same license as the base model Qwen/Qwen3-0.6B. Quantized by André Acioli.
Qwen/Qwen3-0.6B