Qwen3.5-9B GGUF Quant Lab
Reproducible llama.cpp GGUF quantization ladder for Qwen/Qwen3.5-9B, evaluated on English and Traditional Chinese perplexity, TMMLU+, throughput, and VRAM.
This repository contains the text-only language model. It does not include a vision projector (mmproj) and cannot provide the original model's image-input capability.
Recommended default
IQ4_XS-imatrix is the measured sweet spot on the RTX 4090 test system: 4.840 GiB, English/Traditional-Chinese PPL increases of +1.50%/+1.95% from F16, and a TMMLU+ change of -1.0 percentage point. Choose Q4_K_M-imatrix when you prefer slightly stronger PPL retention at a larger 5.243 GiB size.
Quantization and evaluation
Hardware: RTX 4090 24 GiB, Intel i7-13700, WSL2 Ubuntu, CUDA 13.1, llama.cpp b10031. Performance is hardware-specific. PPL is a next-token likelihood measure, not an instruction-following or safety score. TMMLU+ may have appeared in the base model's training data; here it is used only for relative quantization degradation because every quantization receives the identical fixed 500-question sample.
| Quant | GiB | English PPL | 繁中 PPL | TMMLU+ | pp512 tok/s | tg128 tok/s |
|---|---|---|---|---|---|---|
| F16 | 16.690 | 8.0111 | 10.0015 | 68.8% | 7589 | 212 |
| Q8_0 | 8.873 | 8.0190 | 9.9952 | 69.2% | 9038 | 85 |
| Q6_K | 6.854 | 8.0160 | 10.0306 | 69.4% | 7545 | 99 |
| Q5_K_M | 6.024 | 8.0591 | 10.1387 | 68.2% | 8704 | 114 |
| Q4_K_M | 5.243 | 8.1110 | 10.2830 | 68.4% | 8881 | 123 |
| Q3_K_M | 4.306 | 9.0446 | 11.1233 | 63.0% | 8140 | 141 |
| Q2_K | 3.564 | 100.8741 | 44.6410 | 28.6% | 7412 | 159 |
| Q4_K_M-imatrix | 5.243 | 8.0865 | 10.1508 | 68.2% | 8777 | 101 |
| IQ4_XS-imatrix | 4.840 | 8.1311 | 10.1969 | 67.8% | 9539 | 135 |
Full methodology, paired bootstrap CIs, Q2_K regression examples, figures, and reproducibility metadata are in EVAL_REPORT.md.
llama.cpp
./llama-cli -m Qwen3.5-9B-IQ4_XS-imatrix.gguf -ngl 99 --jinja --reasoning off -p "請用繁體中文介紹 GGUF 量化。"
Ollama
Download a GGUF file and the included Modelfile, then run:
ollama create qwen35-9b-quant-lab -f Modelfile
ollama run qwen35-9b-quant-lab
The supplied Modelfile points at Qwen3.5-9B-IQ4_XS-imatrix.gguf. Change FROM to select another local GGUF.
LM Studio
Download one .gguf file, open LM Studio's local model picker, and select the file. This is a Qwen3.5 text-only model; keep the embedded GGUF chat template enabled and disable reasoning if you want the same evaluation-style behavior.
Attribution and license
Base model: Qwen/Qwen3.5-9B at revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The original model and these quantized derivative weights are distributed under Apache-2.0; see LICENSE. Project scripts and documentation are MIT-licensed. This repository was staged for steven0226/Qwen3.5-9B-GGUF-Quant-Lab.