Qwen3.5-4B — native Q4_0 QAT for llama.cpp
Native Q4_0 QAT for Qwen3.5-4B: 17–22% lower mean KL than the tested bartowski and Unsloth Q4_0 builds (91.3% top-1 agreement vs 90.4% / 90.2%), 35% lower than the other QAT Q4_0 tested (YoozLabs, 88.4%), and it beats a larger Q4_K_M of the BF16 original: 0.235 vs 0.319 KL.
2.70 GB · 200/200 decoder projections trained directly on Q4_0 · trained codes packed without re-rounding · vision projector · optional MTP · stock llama.cpp / Ollama / LM Studio.
QUASAR family: NVFP4 W4A16 / vLLM · NVFP4 W4A4 / Blackwell · Q4_0 GGUF / llama.cpp (this repo) · collection · QUASAR paper
Run it
ollama run hf.co/QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF:Q4_0
llama-cli -hf QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF:Q4_0
LM Studio: search QUASAR-QAT Qwen3.5-4B. For image input, load mmproj-Qwen3.5-4B-F16.gguf alongside (llama-server -m Qwen3.5-4B-QUASAR-Q4_0.gguf --mmproj mmproj-Qwen3.5-4B-F16.gguf).
Files
| File | Size | What |
|---|---|---|
Qwen3.5-4B-QUASAR-Q4_0.gguf |
2.70 GB | Default. 200/200 decoder projections Q4_0 (trained), Q8_0 embeddings |
Qwen3.5-4B-QUASAR-Q4_0-MTP.gguf |
2.94 GB | Same, plus the original MTP drafter block for llama.cpp builds with MTP speculative decoding |
Qwen3.5-4B-QUASAR-Q4_0-f16embd.gguf |
3.30 GB | Same weights, F16 embeddings |
mmproj-Qwen3.5-4B-F16.gguf |
0.67 GB | Vision projector from the original model |
Same-harness Q4_0 comparison
llama-perplexity --kl-divergence against the BF16 original (F16 GGUF) on 774 held-out prompts rendered with the chat template, 4096-token chunks. Lower KL and higher top-1 agreement = closer to the original. Full evaluation and reproduction details → EVAL.md.
| GGUF | Size | Mean KL ↓ | Median KL ↓ | Top-1 ↑ |
|---|---|---|---|---|
| QUASAR Q4_0 | 2.70 GB | .235 | .0053 | 91.3% |
| bartowski Q4_0 | 2.78 GB | .284 | .0075 | 90.4% |
| Unsloth Q4_0 | 2.58 GB | .300 | .0077 | 90.2% |
| YoozLabs QAT Q4_0 (QAT on a group-64 lattice, re-quantized to Q4_0) | 2.54 GB | .363 | .0147 | 88.4% |
Q4_K_M, BF16 original (llama-quantize) |
2.78 GB | .319 | .0072 | 90.3% |
plain Q4_0 rounding (llama-quantize) |
2.61 GB | .372 | .0108 | 89.0% |
QUASAR has the lowest KL and highest top-1 agreement of the Q4_0 files tested — PTQ and QAT alike — and also beats the larger Q4_K_M reference while using 3% fewer bytes. Unlike the other QAT file tested, which was trained on a group-64 lattice and then re-quantized to Q4_0, QUASAR is trained directly on the Q4_0 lattice and packs the trained codes without a second rounding pass.
What is quantized
Q4_0 on all 200 decoder projections; Q8_0 embeddings in the default file; norms, conv1d kernels and small delta-net gate vectors remain F16. The GGUF stores QUASAR's trained INT4 codes and scales directly — no llama-quantize re-rounding — and the packed blocks were verified byte-for-byte against the training export.
Training
The 200 decoder projections were trained directly on the Q4_0 lattice with QUASAR using self-distillation from the frozen BF16 model. Training reduced teacher KL from .0383 with plain Q4_0 rounding to .0142.
Citation
@article{counathe2026quasar,
title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
journal={arXiv preprint arXiv:2608.13966},
year={2026}
}