Spark-X2.6-4B — ROCmFPX GGUF
XHToken/Spark-X2.6-4B quantized to two ROCmFPX types. ROCmFPX is a llama.cpp fork with AMD-oriented low-bit weight types for Vulkan and HIP/ROCm.
These files only load in the ROCmFPX fork. Stock llama.cpp, LM Studio and Ollama do not know these types.
| File | Size | |
|---|---|---|
Spark-X2.6-4B-Q8_0_ROCMFPX.gguf |
4.2 GB | 8-bit, closest to the original |
Spark-X2.6-4B-Q4_0_ROCMFP4.gguf |
2.8 GB | 4-bit, smaller and faster |
Measured quality
Against the BF16 model on WikiText-2 test, 100 chunks of 512 tokens, CPU backend. KLD is the mean KL divergence of the token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16. The stock types, made the same way, are there for comparison and are not published here.
| Model | PPL | KLD | Same top |
|---|---|---|---|
| BF16 (original) | 23.0696 | 0 | 100% |
Q8_0_ROCMFPX |
22.6619 | 0.0208 | 93.49% |
Q4_0_ROCMFP4 |
22.4672 | 0.1267 | 84.72% |
Q8_0 (stock, comparison only) |
23.2565 | 0.0049 | 97.01% |
Q4_K_M (stock, comparison only) |
28.5549 | 0.2104 | 80.13% |
How to read this: a perplexity a little below the original's does not mean a quant is better; KLD and same-top measure
how closely it follows the original, and there Q4_0_ROCMFP4 is clearly closer than stock Q4_K_M at a similar
size. Q8_0_ROCMFPX stays close but is further from the original than stock Q8_0. All five files answered the
chat test the same way.
How this was made
- BF16 GGUF from upstream llama.cpp's
convert_hf_to_gguf.py(f849095). The ROCmFPX fork (fb08d7c) runs thespark2_5architecture but its own converter does not have it yet; the fork loads the upstream file as is. - Importance matrix from the original weights on 200 chunks of WikiText-2 train (not the test text). The importance matrix and the BF16 reference row ran on an F32 copy of the BF16 file: the same values, only faster on the CPU.
llama-quantize --imatrixfrom ROCmFPX (fb08d7c), from the BF16 weights, one run per type.- Perplexity and KL divergence on the CPU backend; each file answered a short chat prompt correctly before upload.
Run
git clone https://github.com/charlie12345/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
cmake --build build --target llama-server -j
build/bin/llama-server -m Spark-X2.6-4B-Q4_0_ROCMFP4.gguf -ngl 999 -fa on --jinja
The chat template is the model's own (thinking on by default; --chat-template-kwargs '{"enable_thinking": false}'
turns it off).
License
Apache-2.0, same as the base model. All credit for the model goes to XHToken.