K2-Horizon-MoVA-36B-A4B — APEX GGUF

Imatrix-guided APEX quantization of IFM/K2-Horizon-MoVA-36B-A4B — MBZUAI's Institute of Foundation Models (the LLM360/K2 lineage), released 2026-09-01, Apache-2.0.

Not related to Moonshot's Kimi K2. Coincidental name collision: MBZUAI's K2 lineage (K2-65B with LLM360, 2024) predates Kimi K2 and shares nothing architecturally — no MLA here, and MoVA does not exist in Kimi K2.

⚠️ Requires a forked llama.cpp — upstream CANNOT load this model

Stock llama.cpp will fail to load these files. As of 2026-09-04 upstream has zero support for this architecture (0 hits for k2_horizon / K2Horizon / mova at origin/master 64a155d24). The base model's card claims "PR to llama.cpp is in progress"; no such PR exists on GitHub.

You need this fork and branch:

👉 MBZUAI-IFM/llama.cpp — branch model/K2Horizon

git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DGGML_CUDA=ON          # drop -DGGML_CUDA=ON for a CPU-only build
cmake --build build -j --config Release

Verified at commit 35999d101 (build b10671) — five commits on top of upstream, a textbook new-arch addition (conversion, hparams/tensor loading, compute graph, graph adjustment + tokenizers, chat template). It is 145 commits behind upstream, and almost nothing relevant is missing: tools/imatrix 0 commits, tools/perplexity 0, src/llama-quant.cpp 1 (a threading fix, not a correctness one).

Every llama-server / llama-perplexity / llama-quantize invocation on this page means the binary from that build tree.

Architecture — and why it needed special handling

arch             k2-horizon      798 tensors, 37.445 B params
layers           48, first 3 dense
FFN MoE          100 experts, top-8, +1 shared; expert ffn 768; sigmoid gating, scale 2.5
MoVA             64 VALUE experts, top-4, on 45 of 48 layers
attention        32 heads / 8 KV, head_dim 128
context          524,288          vocab   250,624

MoVA — Mixture-of-Values Attention — is the unusual part. The attention values are a 64-expert routed bank (attn_v_exps [2560, 1024, 64], 7.55 B params = 20.2% of the model), gated by attn_v_gate [2560, 64].

That gate is the reason this quant exists in this form. attn_v_gate is a router, and the shape says so — 64 output columns for 64 experts, structurally identical to ffn_gate_inp [2560, 100] for 100 FFN experts. Its logits pick experts by top-k, so weight error flips a discrete choice rather than adding smooth noise.

The trap: llama.cpp arch-refuses ffn_gate_inp by name, but does not know this name — so a recipe request is honoured and the router really does ship quantized. Left to a generic attn_* rule it lands on a Q3_K floor. Here it is pinned to F32, verified in the shipped bytes (attn_v_gate types: {F32: 45}, quantised-low count 0), at a cost of 24.2 MiB on a 21 GiB file.

Generalizable lesson: any architecture that names its router something new is unprotected by name-based handling. This is the second such case we've hit, after Flash-Next's per_layer_token_embd.

Block legality is completely clean — every mass-carrying ne0 (2560, 1024, 768, 4096, 6144) is 256-divisible, so no QK_K type is illegal and no block-32 substitution occurs.

Files

Compression 3.31× / 4.18× / 4.70×. All three PPL arms were measured back-to-back on the same binary, same corpus and the same offload config, so the tier-to-tier steps below are directly comparable rather than assembled from separate sessions.

Which tier to take — compact is the value pick

The step costs are very unequal, and the agentic suite cannot see this at all (all three tiers score 142/142). Perplexity can:

compact buys 4.36 GiB for a perplexity change that is not statistically resolvable (0.63 σ). Going the next rung to mini costs 7× more perplexity per GiB saved for less than half the space. So:

• compact (16.69 GiB) — take this unless you need the last 1.8 GiB. Best value by a wide margin.
• i-quality (21.05 GiB) — take it if you have the RAM and want the smallest measurable loss.
• mini (14.86 GiB) — only when 1.8 GiB genuinely decides whether the model runs. Its +3.69% vs bf16 is the first tier here whose loss is clearly resolvable (3.93 σ), i.e. the IQ2_S middle band does cost something real even though every test still passes.

For scale on what "resolvable" means here: re-measuring the same file twice on this box moves PPL by 0.0018, so the compact→mini step (+0.1243) is ~69× the measurement noise floor and is a real effect, whereas i-quality's own +1.31% against bf16 sits at 1.42 σ and does not exclude zero.

How the three tiers differ

All three pin the MoVA router (attn_v_gate) at F32 and keep ffn_gate_inp protected. The budget difference lands almost entirely on the routed expert bands:

mini runs the middle band — 57% of the model — at 2.5 bpw and still passes every test case. That is MoE redundancy working as advertised: 100 routed experts at top-8 absorb 2-bit noise that destroys a small dense model at the same width. But passing is the entry bar, and perplexity shows the 2-bit band is not free: mini is the only tier here whose loss against bf16 is clearly resolvable. If you need smaller, the next rung down (IQ2_XXS, ~2.06 bpw) is where we stopped — untested, and given how sharply the cost rose from IQ3_XXS to IQ2_S, not recommended blind.

compact and mini were built from a 310-chunk code-inclusive imatrix (uploaded as k2-horizon-36b-calib2.imatrix) rather than i-quality's 120-chunk prose one, because calibration corpus choice is known to move routed experts much more than attention — which is exactly what these tiers quantise hardest. At i-quality's 4.83 bpw the same swap measured as a null (limitation 3).

Note that i-quality carries a coarser allocation than the smaller tiers: its budget lands on Q6_K/Q6_K/Q3_K with nothing in between, while compact and mini spread smoothly. See limitation 1.

Read the confidence interval, not the point estimate

delta           +0.0929 PPL  (+1.34%)
combined sigma   0.0641  ->  1.45 sigma
95% CI          [-0.47%, +3.15%]

The CI includes zero, so at 137 chunks this corpus cannot resolve the quantization cost. That is not a null result — quantization always costs something — it means the cost is small relative to the measurement noise floor. Tightening it needs a larger eval corpus, not more --chunks: 137 × 2048 = 280,576 tokens is everything wiki.test.raw yields with this tokenizer.

This delta is measured against our own bf16 — same box, same binary, same corpus, same tokenizer, no borrowed baseline and no cross-machine term. That is only possible because the bf16 fits in host RAM (69.78 GiB), which it does not for most models this family's size.

Corpus pinned: WikiText-2 raw test split, 1,290,590 bytes, sha256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.

Do not compare 7.0270 to other models' PPL. This tokenizer (vocab 250,624) needs 137 chunks where Qwen's needs 145 on the identical corpus; higher per-token perplexity is the arithmetic consequence of packing more information per token, not a quality signal. The percentage cost is the more portable quantity.

Agentic capability — 42/42 tasks, 253/253 test cases

Real agentic coding suites: the model drives an actual tool-use loop (read/write/run files) and is graded by a fresh pytest it never sees, at temp 1.0. Full sweep on a Xeon + RTX 4060 Ti 16 GB:

Perfect score, zero failures, zero context truncations. Single-slot throughout (n_slots=1, kv_unified=false, n_ctx_slot=49152, verified in the server log, not assumed).

All three tiers, frontier-9 (the hardest suite)

A full score here is the entry bar, not the result — this suite is built to saturate for a competent coder, so 142/142 means the tier qualified, and it is not evidence that dropping from 4.83 to 3.41 bpw costs nothing. The discriminating metric at equal score is output tokens, and at one run per tier it cannot separate them: across these four runs of near-identical models, btree_insert_delete alone spans 1,731 to 79,583 tokens (46×) and six of nine tasks span more than 4×. So the token column is published as absolute per-run numbers, and no ratio between tiers should be read off it. Separating the tiers on cost would need >=3 reps each.

An earlier frontier-9 run on a DGX Spark (GB10) scored 8/9 (139/142) — the single difference was one flip-prone task where that run hit a DP reconstruction off-by-one. Same model, same quant, same commit. That is run-to-run variance, not a finding about either box, and it is the same effect the token spread above shows.

Running it

⚠️ These commands need the FORKED binary — stock llama.cpp cannot load these files

If you scrolled straight here to copy a command, read this first. k2_horizon / mova has zero support in upstream llama.cpp, so a stock llama-server will refuse the file at the architecture check. Build the fork:

git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp llama.cpp-k2horizon
cd llama.cpp-k2horizon && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

Full detail, verified commit and what the branch changes: see the top of this card. Every llama-server, llama-perplexity and llama-quantize invocation below means that binary, which is why the path is spelled out rather than assumed to be on your PATH.

./llama.cpp-k2horizon/build/bin/llama-server \
  -m K2-Horizon-MoVA-36B-A4B-APEX-i-quality.gguf \
  --ctx-size 32768 --parallel 1 \
  -ngl 99 -ncmoe 40 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -fa on --jinja \
  --reasoning-format deepseek --reasoning on --reasoning-effort high

--n-cpu-moe does not move the MoVA bank. It knows FFN experts only, so attn_v_exps stays on the GPU unless you ask with -ot "attn_v_exps=CPU". Counter-intuitively the default is the fast one: keeping that bank on the card is worth +30% generation (13.5 vs 10.4 t/s), and at this quant it is 4.06 GiB and fits — at bf16 it was 14.06 GiB and had to be offloaded.

Generation speed on a 16 GiB RTX 4060 Ti + 20 CPU threads (-ncmoe N = layers whose FFN experts stay on CPU, so 48 = all of them):

-ncmoe 32 is fastest but not the one to run — 719 MiB of headroom on a 16,384 MiB card, and ~2 GiB of what is already there is the desktop. Anything that grows the desktop's footprint mid-run turns a working config into a cudaMalloc failure. -ncmoe 40 trades ~2 t/s for ~3 GiB of slack. Identical invocations measured 18.3 then 17.5 t/s, so treat one-decimal differences as noise. These are single-stream, empty-cache figures.

KV cache is 192 KiB/token at f16 (48 × 8 × 128 ÷ 256) — 24 GiB at 128k, 96 GiB at the advertised 512k, so the full context is not reachable on a workstation. Use q8_0 KV.

It is a reasoning model. The template uses <ifm|think>; llama-server's deepseek extractor handles the non-standard tag correctly (reasoning_content populated, content clean, no leak). Reasoning at effort=high is heavy — 168 tokens for "17*23" — so give it a generous output budget or you get empty content with finish_reason=length.

Honest limitations

Three, all specific and all fixable — recorded because a quant card that only lists strengths is not useful.

1. The allocation is a cliff, not a gradient. The histogram is {Q8_0: 144, Q6_K: 265, F32: 45, Q3_K: 112} — nothing between Q6_K and Q3_K. The middle expert band (21.2 B params, 57% of the model) dropped five menu steps in one move while 12.9 B stayed at Q6_K, because structural allocation steps whole bands and one step less would have overshot the target. A measured per-tensor allocation would smooth this; this architecture does not have a sensitivity name-map yet. (Allocator accuracy itself was good: predicted 22.59 GB, artifact 22.601 GB.)

2. attn_v_exps placement is reasoned by analogy, not measured. It is treated as a routed expert bank on its access pattern (sparse 4-of-64), which is the defensible default. But a KDA census found value tensors the most sensitive attention projection, and those were a single dense V seeing every token where each expert here sees ~1/16 of traffic. At 20.2% of the model this deserves a sensitivity pass before anyone calls it measured.

3. The calibration is thin, and for MoVA a bigger corpus does not fix it. This tier was built from a 120-chunk imatrix (61,440 tokens) on a general/scientific-prose corpus, because that corpus — not --chunks — was the binding limit.

   A second imatrix was then built on a 2.6× larger, code-inclusive corpus (310 chunks, 158,720 tokens). Comparing the two directly, on the accumulated .counts in each file:

   	FFN experts (ffn_{up,down}_exps)	MoVA experts (attn_v_exps)
   120-chunk prose	0 unrouted, min 1 route	6 unrouted of 2,880
   310-chunk mixed	0 unrouted, min 3 routes	the same 6 unrouted

   The unrouted set is identical — blk.{35,39,40,41,42,44}.attn_v_exps, experts 60/15/41/24/10/45 — across two corpora that differ by a whole modality (the first contains no code at all). And there is no gradual tail behind them: those 6 cells are exactly 0 while the 1st percentile of all others is 114 routes and the median is 7,274. Only 6 of 2,880 cells sit below 10 routes, and they are the same 6.

   So these read as structurally unselected experts, not a sampling gap. Stated as measured: neither corpus ever routes them. Both are English-dominant, so a genuinely different input distribution has not been ruled out. Six of 2,880 cells (0.2%) have no importance data.

   And the larger corpus does not improve perplexity either — measured. A second tier was built from the 310-chunk imatrix with a byte-identical allocation (same recipe, same type histogram), so calibration data was the only variable:

   arm	PPL
   this file (120-chunk imatrix)	7.0252 +/- 0.04550
   rebuild (310-chunk imatrix)	7.0269 +/- 0.04550

   +0.0017, i.e. 0.026 sigma — indistinguishable, and nominally the wrong direction. The control that makes it conclusive: re-measuring this same file moved it 0.0018 from its own earlier 7.0270, so the same-file noise floor is 1.06x the effect being tested. (Likely mechanism: MoVA's top-4 routing turns a 1-ULP difference into a different expert selection.) The rebuild also cleared all 42 agentic tasks (253/253), which is the suites' entry bar rather than a null result, and its perplexity is inside the same-file noise floor. It is not published as a separate tier on that basis. (Tokens-at-equal-score, the discriminating agentic metric, was not run to sufficient reps to separate them either way.)

   Consequence: calibration is empirically ruled out as the lever at this size point, which promotes limitation 1 (the allocation cliff) to the prime suspect for the +1.34%.

Provenance

Built with quantkit dynamic_apex.py; the MoVA classifier rules landed as 9e91f50. Verified before building: 13 assertions against the real tensor list pass, and a regression over 294 name/layer combinations from other supported architectures shows zero movement — the rules anchor on attn_v_gate.weight$ / attn_v_exps.weight$, which no other architecture has.

PPL measured on the fork binary for both arms, which makes any kernel offset common-mode and cancels it in the delta.