Nanbeige4.3-33B — ROCmFPX GGUF

Nanbeige/Nanbeige4.3-33B, Nanbeige's dense 33B agentic coding and reasoning model, quantized to the ROCmFPX formats, a llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

Start with Nanbeige4.3-33B-Q6_K_ROCMFPX.gguf (27.3 GB) on a Strix Halo or a 32 GB+ GPU. This model loses a lot more at 4-bit than most: both ROCmFPX 4-bit files are clearly further from BF16 than a stock llama.cpp Q4_K_M (see below). If you need a file under 20 GB and do not need the ROCmFPX types, use a stock Q4_K_M.

These files need HarshaVardhan3002/ROCmFPX. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds may not load them. These files have not been tested on AMD hardware yet.

Not an official release of Nanbeige. All credit for the model goes to its authors.

Files and measured quality

Reference: the BF16 model (perplexity 21.52). Measured on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the next-token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16. Perplexity on raw Wikipedia text is high for this chat-tuned model and moves by less than its error margin (about ±0.8) between files, so KLD is the number to compare.

The 6-bit file is named Nanbeige4.3-33B-Q6_K_ROCMFPX.gguf so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock llama.cpp Q6_K, and it needs the same fork as the other files.

For comparison, a stock llama.cpp Q4_K_M of the same model (about 20 GB), scored the same way, measured KLD 0.0716, same top 89.97%. That is clearly closer to BF16 than both ROCmFPX 4-bit files.

Keeping the output layer at 8-bit (the _O8 file) lowers KLD from 0.1744 to 0.1114 for a slightly larger file. Also measured, not published: plain Q4_0_ROCMFP4_STRIX (KLD 0.1744), the FAST layout with an 8-bit output layer (0.1169, no better than the _O8 file), STRIX with attention and output at 8-bit (about 22 GB, 0.0824, larger and further from BF16 than stock Q4_K_M), and Q3_0_ROCMFPX (0.2193, larger and further from BF16 than the 4-bit files).

How these were made

1. BF16 GGUF converted with the fork's convert_hf_to_gguf.py.
2. Importance matrix from the BF16 file over a general calibration text (calibration_datav3), 512-token chunks.
3. llama-quantize --imatrix from the BF16 file, one run per type, from HarshaVardhan3002/ROCmFPX; the _O8 file adds --output-tensor-type q8_0_rocmfpx. No tensor scales were needed (the model's weights are large enough for the ROCmFPX block scales).
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock kernels. Each quant was decoded by the fork's CPU code to F16 and scored on the same GPU, so the numbers measure the weights only. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m Nanbeige4.3-33B-Q6_K_ROCMFPX.gguf -ngl 999 -fa on --jinja -c 32768 --temp 0.6 --top-p 0.95

Nanbeige recommends temperature 0.6 / top-p 0.95 for reasoning and chat, and 1.0 / 0.95 for agentic and tool use. The model thinks before it answers (<think> block) by default; the chat template's enable_thinking=false turns that off. The original supports up to 512K tokens of context.

License

Apache-2.0, same as the base model (see LICENSE).