Qwen3.5-4B fraQtl Q4_0 (NPU-native)

Model Conversion Contributor: fraQtl. Canonical repo and full receipts: https://huggingface.co/fraQtl/Qwen3.5-4B-NPU-GGUF

37.8% closer to the original model on task text and 28.4% closer on general text than the stock unsloth Q4_0, at 41,024 fewer bytes. Same speed on the Snapdragon 8 Elite Gen 5 NPU, with every operation on the NPU.

Fidelity to the original (bf16)

Full-vocabulary KL divergence to bf16 Qwen/Qwen3.5-4B over 144 held-out windows. Lower is closer. 95% CI from a paired bootstrap (2,000 resamples).

Task = instruction following + tool calling. General = knowledge + reasoning. Against AutoRound Q4_0 (28.6 MB larger): task −20.5% [−24.3, −16.6], general −3.4% [−10.4, +5.0].

Other 4-bit files at this size

Same 144 windows and scorer. The last two columns compare each file with the fraQtl Q4_0 file; a negative number means that file is closer to the original.

unsloth Q4_K_S (7.25 MB larger) is 16.0% closer on general text, and within 1.6% on task text with the interval crossing zero. unsloth IQ4_NL (3.2 MB smaller) is 7.3% closer on general text and 13.4% further on task text. Both use only Hexagon-supported types; neither was run on the phone here.

Answers on 500 fixed questions

Greedy decoding, thinking off, both files on the same GPU in the same run.

Differences: GSM8K 5 right only on each file (p = 1.00); IFEval 14 vs 10 (p = 0.54). bf16 reference: 232 / 195.

On device: Snapdragon 8 Elite Gen 5 (SM8850), llama.cpp b12079 Hexagon backend

File sha256 verified on the device. pp128 = prompt processing, tg64 = generation, tokens/s, 3 repetitions.

Method

1. Rounding: Hessian-weighted GPTQ rounding (Frantar et al., 2022) with statistics from our calibration corpus.
2. Allocation: Q4_0 for most weights; 52 tensors at Q8_0 inside the byte budget. 47 of those (24 ssm_beta, 23 ssm_alpha) follow unsloth's own choice; 5 were picked by our per-tensor error ranking (blk.30 and blk.31 ffn_down, blk.23 and blk.27 attn_k, blk.30 ssm_out). The unused next-token-prediction block is dropped.
3. Recovery training was tested on this file and gave no gain (one seed +4.4% on task KL, one unchanged), so it is not applied.
4. Types in the file: Q4_0 192 · Q8_0 52 · Q4_1 4 · Q6_K 1 · F32 177. All matmul types are Hexagon-supported; the file loads in stock llama.cpp.

Scope

• 144 KL windows and 500 questions; one build of each file. KL measures closeness to the original model, not task accuracy.
• Text model only.
• Comparators: unsloth Q4_0, IQ4_NL and Q4_K_S; Intel AutoRound Q4_0.
• Credit: Intel AutoRound for the second comparator; unsloth for the comparator file and the ssm_alpha/ssm_beta precision choice; the Qwen team for the base model; llama.cpp for the format and the Hexagon backend.

Files

Receipts: receipts/ (fidelity table, answers, device log, rejected-op count, Q8_0 tensor list).