Kolibri-1 — ROCmFPX GGUF

Aleph-Alpha/Kolibri-1-BF16, Aleph Alpha's 78B mixture-of-experts reasoning model for German and English (about 3.5B active per token), quantized to the ROCmFPX formats, a llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

Start with Q4_0_ROCMFP4_STRIX on a 128 GB Strix Halo, or Q6_0_ROCMFPX if you have the memory.

These files need HarshaVardhan3002/ROCmFPX at commit b130d4995 or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. These files have not been tested on AMD hardware yet.

Not an official release of Aleph Alpha. All credit for the model goes to its authors.

Files and measured quality

The BF16 model (156 GB) does not fit in the memory of the machine used here, so the reference is a stock llama.cpp Q8_0 of it (83 GB), which is normally very close to BF16. Measured on WikiText-2 test, 16 chunks of 512 tokens, on the fork's CPU backend. KLD is the mean KL divergence of the next-token distribution from that Q8_0 (lower is better). Same top is how often the quant picks the same next token. Only English text was measured.

The 6-bit file is named Kolibri-1-Q6_K_ROCMFPX.gguf so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock llama.cpp Q6_K, and it needs the same fork as the other files.

Not published: Q3_0_ROCMFPX (43.6 GB, KLD 0.1572) is larger than Q4_0_ROCMFP4_STRIX and further from the reference.

In Q8_0_ROCMFPX the token embedding is stored as stock Q8_0 (KLD 0.0682 with it as Q8_0_ROCMFPX).

Why these files need a newer build

Kolibri's weights are much smaller than those of most models. ROCmFPX stores one small scale per block of 32 weights, and that scale has a fixed smallest step, so many blocks would lose most of their precision (ROCmFPX#118).

These files were made with an opt-in fix in llama-quantize (LLAMA_ROCMFPX_TENSOR_SCALE=1): each weight tensor (each expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing (for Kolibri mostly 2^9), and the inverse factor is stored as a small extra .scale tensor that llama.cpp applies to the matmul output. The factor is a power of two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject.

How these were made

1. convert_hf_to_gguf.py --outtype bf16 from the fork, plus a stock Q8_0 copy of it.
2. Importance matrix from the Q8_0 copy on the GPU over a general calibration text (calibration_datav3), 512-token chunks.
3. LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix from the BF16 file, one run per type, from HarshaVardhan3002/ROCmFPX b130d4995.
4. Quality: the Q8_0 reference ran on an NVIDIA GPU with stock kernels; each quant was scored with the fork's CPU backend, which runs the ROCmFPX types directly. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m Kolibri-1-Q4_0_ROCMFP4_STRIX.gguf -ngl 999 -fa on --jinja -c 32768

License

Apache-2.0, same as the base model (see LICENSE). Please also follow Aleph Alpha's responsible-use notes in the original model card.