K2-Horizon-MoVA-36B-A4B GGUF quants

GGUF quants of IFM/K2-Horizon-MoVA-36B-A4B. The base model has 36B total parameters and approximately 4B active parameters per token.

These are straight llama-quantize conversions from the original BF16 GGUF published by IFM. No importance matrix or calibration data was used.

Files

Required llama.cpp build

K2 Horizon architecture support is required. Use the IFM fork/branch until support is available in the upstream version you are using:

git clone --depth 1 --single-branch --branch model/K2Horizon \
  https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

Run a quant with all GPU layers and 36 CPU MoE layers:

./build/bin/llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf \
  -ngl 999 -ncmoe 36

Benchmark

Measured locally on an NVIDIA GeForce RTX 3090 (24,103 MiB) and AMD Ryzen 9 9900X. The benchmark used -p 512 -n 128 -b 512 -ub 512 -r 3 -ngl 999 -ncmoe 36.

The benchmark results and short inference checks were run with the K2Horizon branch and are included alongside this README in the local build logs.

Source

• Base model: IFM/K2-Horizon-MoVA-36B-A4B
• GGUF conversion source: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• Required llama.cpp fork: MBZUAI-IFM/llama.cpp, model/K2Horizon