Qwen3.5-4B — QUASAR NVFP4

Project page · Code

2.3–2.7× lower KL to BF16 than the leading public 4-bit builds we tested. Best 4-bit AIME'25, GPQA-Diamond, MMLU-Pro and GSM8K-Platinum in our same-harness comparison.

4.2 GB · all 200 decoder projections in NVFP4 · vision + MTP retained · 98.2% hard-task recovery · +35% single-stream decode vs BF16 on H100 · serves directly with vLLM.

Also in this family: W4A16 / vLLM (this repo) · W4A4 / Blackwell · Q4_0 GGUF / llama.cpp · collection · paper

A weight-only NVFP4 (W4A16) version of Qwen/Qwen3.5-4B trained with QUASAR quantization-aware training: the quantized weights are learned by distilling from the frozen BF16 original on its own outputs, so this is a drop-in lower-precision replacement, not a fine-tune. Companion to Qwen3.8-27B-QUASAR-NVFP4 in the Qwen3.5-4B & Qwen3.8-27B — QUASAR 4-bit QAT collection.

Same-harness comparison with leading 4-bit builds

One harness for every row, following RedHatAI's published protocol (0-shot chat, thinking on, 65,536-token budget, vLLM 0.26 on H100). AIME'25 is the mean of 8 seeds, GPQA-Diamond of 3. Bold = best among the compared 4-bit builds.

Full benchmark table, seeds, confidence information, fidelity methodology and serving measurements: EVAL.md.

• Closest to BF16. 2.3–2.7× lower per-token KL than the compared public 4-bit builds, with 96.2% top-1 agreement.
• Strongest reasoning results. Best compared 4-bit result on AIME'25, GPQA-Diamond, MMLU-Pro and GSM8K-Platinum; GPQA-Diamond is within seed noise of BF16.
• Full decoder scope, full feature set. All 200 major decoder projections in NVFP4 while preserving the vision tower and MTP drafter; 3.29 GiB of weights in VRAM and +35% single-stream decode over BF16 on H100.

Run it

pip install "vllm>=0.26"

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3

Loads straight from the Hub, no conversion (compressed-tensors NVFP4, weight-only / W4A16). Validated with vLLM 0.26 on H100 for text, image input, 262K context (needle retrieved from a 200k-token prompt) and MTP speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":2}': +27% single-stream decode, 80% draft acceptance). For Blackwell, use the W4A4 build; for llama.cpp / Ollama, the Q4_0 GGUF.

Fidelity vs standard quantization at the same scope

Per-token forward KL(BF16 ‖ quantized) and top-1 agreement on 734 held-out prompts of the BF16 model's own responses (934k response tokens), all methods at the same 200-projection NVFP4 scope (same data and schedule where trained):

At identical NVFP4 scope, QUASAR cuts RTN KL by 2.3× and improves ~20% over both GPTQ and standard QAT.

Quantization details

NVFP4 (E2M1 weights, group 16, FP8-E4M3 group scales under one fp32 tensor scale) on every large projection in every decoder layer: MLP gate/up/down in all 32 layers, q/k/v/o in the 8 full-attention layers, and in_proj_qkv / in_proj_z / out_proj in the 24 gated delta-net layers — 200 of 200. Kept in BF16: embeddings (tied), norms, the 24 depthwise conv1d kernels, the 48 small delta-net gate projections (0.11% of the decoder), the vision tower and the MTP drafter (bit-identical to the original). Full-decoder NVFP4, with vision and MTP retained.

The packed NVFP4 weights decode to exactly the weights that were trained and evaluated (222,822,400 / 222,822,400 groups, 0 flipped elements — export_ct_report.json); fused q/k/v, gate/up and in_proj_qkv/z groups share their global scale by construction, so vLLM's fused-layer loading changes nothing.

Training. QUASAR QAT with self-distillation from the frozen BF16 checkpoint over ~120M tokens; QUASAR performs loss-aware NVFP4 reconstruction inside the training loop. The exported packed weights are bit-exact to those trained and evaluated. Details in EVAL.md and the paper.

Related

• QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 — the 27B sibling: 496/496 linears in NVFP4 (W4A4), GPQA-Diamond 0.909 vs 0.914 BF16.
• QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4-W4A4 — the same weights with calibrated activation scales for native W4A4 on Blackwell.
• QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF — native Q4_0 QAT for llama.cpp / Ollama / LM Studio.
• Qwen3.5-4B & Qwen3.8-27B — QUASAR 4-bit QAT — the collection; Gemma 4 and Muse Glimmer builds under QUASAR-QAT.
• Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction.

Citation

arxiv.org/abs/2608.13966

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}