Qwen3.5-4B — ROCmFP4 / ROCmFPX GGUF
First ROCmFP4/ROCmFPX quantization of Qwen/Qwen3.5-4B. Four quants, built from the Hub
BF16 GGUF (no re-conversion), each individually verified on real hardware.
Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified — with the ROCmFPX llama.cpp fork.
⚠️ Needs a ROCmFPX-capable llama.cpp build. These will not load in stock llama.cpp / Ollama / LM Studio.
Variants
| file | ftype | size | token_embd |
decode | correctness |
|---|---|---|---|---|---|
Qwen3.5-4B-Q4_0_ROCMFP4_COHERENT.gguf |
102 | 2.37 GiB | Q6_K | 55.57 t/s | 3/3 |
Qwen3.5-4B-Q6_0_ROCMFPX_AGENT.gguf |
114 | 3.67 GiB | Q8_0 | 37.13 t/s | 3/3 |
Qwen3.5-4B-Q8_0_ROCMFPX.gguf |
111 | 4.07 GiB | Q8_0 | 37.25 t/s | 3/3 |
Qwen3.5-4B-Q8_0_ROCMFPX_AGENT.gguf |
115 | 4.12 GiB | Q8_0 | 36.92 t/s | 3/3 |
mmproj-BF16.gguf is included — required for image input (-fa off).
Which to pick: the 4-bit (102) — fastest and smallest, same answers as the 8-bits.
⚠️ Q6_0_ROCMFPX_AGENT (114) is the weakest choice here: larger than the 4-bit and
slower than the plain 8-bit. The AGENT recipe raises speculative-draft acceptance, and
Qwen3.5-4B ships no drafter, so that benefit cannot apply. It is included for completeness.
Measured — not estimated
Hardware: Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified.
-ngl 999 -c 4096 -fa on -fit off. 300 tokens, warm-up discarded, median of 3.
| ftype | run 1 / 2 / 3 | median | spread |
|---|---|---|---|
| 102 | 55.58 55.57 55.56 | 55.57 | 1.0 |
| 114 | 37.15 37.13 37.13 | 37.13 | 1.001 |
| 111 | 37.2 37.25 37.41 | 37.25 | 1.006 |
| 115 | 36.85 36.92 36.97 | 36.92 | 1.003 |
No MTP/EAGLE drafter ships with this model, so these are the deployed numbers with nothing left on the table.
Head protection
Qwen3.5-4B has tied embeddings — there is no output.weight tensor, so
--output-tensor-type is a silent no-op and --token-embedding-type is the only
flag that protects the head. Audited by exact tensor name on every artifact.
2543899552 Qwen3.5-4B-Q4_0_ROCMFP4_COHERENT.gguf
3935341472 Qwen3.5-4B-Q6_0_ROCMFPX_AGENT.gguf
4370869152 Qwen3.5-4B-Q8_0_ROCMFPX.gguf
4426410912 Qwen3.5-4B-Q8_0_ROCMFPX_AGENT.gguf
Credits
- Base model: Qwen/Qwen3.5-4B
- Quantization stack: ROCmFPX