K2-Horizon-MoVA-36B-A4B — GGUF

GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, produced with the MBZUAI-IFM fork of llama.cpp (model/K2Horizon branch) that adds native k2-horizon architecture support.

Architecture note: Upstream llama.cpp does not yet support the k2_horizon / K2HorizonForCausalLM architecture (PR in progress as of September 2026). You must use the MBZUAI-IFM fork linked above — or upstream once the PR merges — to load these GGUFs.

---

Model Overview

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family released by the Institute of Foundation Models (IFM) at MBZUAI on 3 September 2026. It is a Mixture-of-Experts (MoE) model augmented with a novel Mixture-of-Values (MoVA) attention mechanism, storing ~36 B parameters while activating only ~4 B per token.

---

What Is MoVA?

Mixture-of-Values (MoVA) applies the same sparse routing philosophy as MoE, but inside the attention mechanism: the value projections are replaced by a learned pool of value experts, and only a small subset is activated per token. This lets the model store a much richer set of representational "styles" in the value space without paying for all of them at inference time, complementing the feed-forward MoE sparsity.

---

Quantization Details

IQ quants (those beginning with IQ) were generated using an importance matrix computed on the Q2_K quant with 8 × 512-token chunks of WikiText-2 calibration data (Salesforce/wikitext).

All quants were produced from the BF16 GGUF published by IFM at IFM/K2-Horizon-MoVA-36B-A4B-GGUF. No safetensors conversion was performed; the IFM BF16 GGUF already includes tokenizer metadata and a llama.cpp-compatible chat template.

---

Hardware Requirements (Rough Guide)

These figures are weight-only estimates and exclude KV cache, runtime overhead, and context length. They are not minimum VRAM/RAM requirements.

Memory caveat: "4B active" means 4B parameters are computed per token, not that the model fits in 4B-worth of memory. All 36B weights must be resident in RAM/VRAM.

---

Running with llama.cpp

Prerequisite: build from the MBZUAI-IFM fork (model/K2Horizon branch) until upstream llama.cpp merges the PR.

git clone --depth 1 --branch model/K2Horizon \
    https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DGGML_NATIVE=OFF
cmake --build build -j4

Basic inference:

./build/bin/llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -p "Explain the difference between MoE and MoVA architectures." \
  -n 1024 \
  --temp 1.0 \
  --top-p 0.95 \
  -c 8192

---

Serving (Original BF16 Model)

The original BF16 safetensors checkpoint can be served with vLLM or SGLang.

vLLM:

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

SGLang (validated on 2 × H200):

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

---

Best Practices

1. Always use reasoning_effort="high" — all reported benchmark scores were obtained at high reasoning effort.
2. Recommended sampling: temperature=1.0, top_p=0.95, max_new_tokens ≥ 32 768.
3. Tool use / agent tasks: enable the k2_horizon tool-call parser when using the vLLM or SGLang API.
4. llama.cpp context: the model supports 512 K natively, but local inference is memory-constrained — use -c 8192 or -c 32768 unless you have sufficient RAM.

---

Benchmark Results

Results from the IFM model card. All scores in %. Bold = best in row. K2-Horizon-MoVA-36B-A4B uses high reasoning effort; other open models are in their respective reasoning modes.

---

Source & Links

• Original model: IFM/K2-Horizon-MoVA-36B-A4B
• Official BF16 GGUF: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• llama.cpp fork (K2-Horizon support): MBZUAI-IFM/llama.cpp @ model/K2Horizon
• IFM blog post: https://ifm.ai/blog/k2/
• vLLM recipe: https://recipes.vllm.ai/IFM/K2-Horizon-MoVA-36B-A4B
• SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/IFM/K2-Horizon

---

Quantization Pipeline

These GGUFs were generated with the following pipeline, mirroring the approach used for other large MoE checkpoints in this workspace:

1. Fork checkout & build — the MBZUAI-IFM model/K2Horizon branch is cloned to a dedicated directory and compiled with CMake (GGML_NATIVE=OFF for portability).
2. Source download — the IFM BF16 GGUF (~70 GiB) is downloaded from IFM/K2-Horizon-MoVA-36B-A4B-GGUF instead of converting from safetensors (a direct BF16 or Q8_0 conversion would exceed the ~109 GiB free on /tmp).
3. Imatrix — computed on the Q2_K quant (~10 GiB, fits in 15 GB RAM) with 8 calibration chunks of WikiText-2 to produce an importance matrix for the IQ quants.
4. Quantization — all standard and IQ quants are produced from the BF16 GGUF source; each file is deleted immediately after a verified upload to keep disk usage under BF16 (~70 GiB) + one quant at any given moment.
5. Resume support — a status.json file tracks which quants have been generated and uploaded so interrupted runs resume from where they left off.

---

License

The original model and these GGUF quantizations are released under the Apache 2.0 License. See LICENSE on the upstream repository for full terms.

---

Citation

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}