primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4

Half the size of BF16. Level on knowledge and abstention, 3.1 behind on call accuracy.

4-bit quantization of IFM/K2-Horizon-MoVA-36B-A4B to 36.74 GB, on one 96 GB card.
The 13,500 routed expert projections are NVFP4; everything else stays BF16.

This is the small, portable one. There is also a mixed NVFP4/FP8 build at 44.48 GB that is 1.7 higher on call accuracy but needs Blackwell. This build is the one that also runs on Ampere, verified generating on an A100 40 GB. Take it for pre-Blackwell hardware, for the smallest footprint, or for throughput under load; take the mixed one if call accuracy matters more than any of those.

---

Why this quant

• 🗜️ 2.0× smaller than BF16, 24% smaller than IFM's own FP8. 36.74 GB against 74.89 GB and 48.36 GB.
• 🎯 Knowledge is level with both references. 86.5 against the official FP8's 86.8 and BF16's 85.9 over 1,170 items.
• 🧭 Abstention is level too, and that is the useful half of the tool-calling result. 60.4 against the official FP8's 60.4, the same number. What 4 bits cost is argument accuracy on calls the model decides to make: 71.8 against 74.9, a real gap at t = −4.2, and two of the five benchmarks carry 90% of it.
• ⚡ 1.25× the official FP8's throughput at concurrency 32, and first token in a third the time. 301.5 against 240.6 tok/s, median TTFT 1.54 s against 4.46 s. Single-stream is a wash, and the throughput section explains why that is architectural.
• 🖥️ One card, no tensor parallelism. Both official recipes use TP=2; this fits TP=1 with 275,952 tokens of KV at 32K context.
• 🧩 3,408 modules stay at BF16. IFM's own FP8 leaves the same set alone. The reason is architectural and spelled out below.
• 🔧 Serves on stock vLLM main, no patches and no custom kernels. Plain compressed-tensors. When this repo first went up, on 4 September 2026, no NVFP4 build of this model served on vLLM.

---

Serve it

K2-Horizon support merged into vLLM main on 3 September 2026 (#55063) and is not in a release yet — 0.28.0 predates it. Use a main/nightly build:

hf download primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4 \
  --local-dir ./K2-Horizon-MoVA-36B-A4B-NVFP4

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c \
  --model /models/K2-Horizon-MoVA-36B-A4B-NVFP4 \
  --trust-remote-code --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser k2_horizon

That image tag is the nightly every number on this page was measured on (vllm 0.28.1rc1.dev388). On one RTX PRO 6000 Blackwell it leaves 34.87 GiB resident for weights and 50.53 GiB for KV — 275,952 tokens at 32K context, 8.42× concurrency — and the engine is up in 97 s, of which 3 s is torch.compile.

Reasoning effort is a server-level choice on this architecture, not a per-request one. The template emits a different tag pair per effort and K2HorizonReasoningParser picks one pair once at startup, defaulting to high:

• high (the default) — <ifm|think> … </ifm|think>
• medium — <ifm|think_fast> … </ifm|think_fast>
• low — <ifm|think_faster> … </ifm|think_faster>

Send a per-request reasoning_effort: low against a server parsing high and the tags never match, so the whole reasoning trace stays in content instead of being separated out. Set the effort on the server. Split reasoning is returned in message.reasoning.

---

Measured

1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite — MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500 — and a 200-item tool-calling suite — BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, When2Call — under one fixed protocol for every row: PQE-1.2, temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, concurrency 32, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.

Auto-scored, no LLM judge. A knowledge item is scored on the last ANSWER: in the reply; a tool-calling item counts only if every expected call is present with acceptable arguments and no call is invented. Sizes are the safetensors weights, and ± is the sample standard deviation over the runs counted in the runs k/a column.

call is accuracy on the 160 tool-calling items that require a call; abstain is the 40 whose correct action is to call nothing; tool calling is the two pooled as 200. We report the halves separately because pooling hides which one moved. runs k/a is the number of knowledge and agentic runs behind each row. finished is the share of the 1,170 knowledge items that answered inside the token budget; out/answer is mean output tokens.

Knowledge is one group. No pair separates: against the official FP8 this build is t = −0.6, the mixed build t = +0.6, BF16 t = −1.5, all at n=2 per side. Don't rank on this column.

Call accuracy is what 4 bits cost this model. 71.8 against the official FP8's 74.9 — 3.1 points, t = −4.2 over six runs each. Abstention is untouched at 60.4 against 60.4, the same number to two decimals, so the model has not become trigger-happy or timid. It picks arguments wrong slightly more often.

Nor is it a formatting failure. Unparsed replies run at 9 per 1,200 here (0.75%) against 13 per 1,200 for the official FP8 (1.08%) and 6 per 600 for BF16 (1.00%) — the lowest rate of any build measured. Thinking fired on 99–100% of items in every row.

Per-benchmark detail, both suites

Pooled over every run behind the row, so a cell rests on runs × items. when2call supplies the 40 abstention items; the other four tool-calling benchmarks supply the 160 call items.

Two benchmarks carry the tool-calling gap. xlam/APIGen falls 44.8 → 37.8 against the official FP8, 7.0 points on 45 items, and toolace 78.1 → 75.2, 2.9 points on 45. Weighted by item count those two are 64% and 26% of the 2.5-point pooled gap; bfcl contributes 0.17 of a point, glaive 0.08, and abstention does not move at all. Both carry the longest argument lists in the suite, which fits 4 bits costing argument accuracy rather than call structure.

The noise floor, and why the run counts are in the table. Every build compared here has at least six agentic runs, the one exception being the BF16 reference at three. It needs to be six. The official FP8's tool-calling figure read 72.7 ±0.3 after three runs and 72.0 ±0.8 after six; this build's read 69.0 ±1.5 after three and 69.5 ±1.4 after six. On the 1,170-item knowledge suite, repeat runs of a single checkpoint span up to 1.9 points, so treat a knowledge gap of a point or so as unresolved. The widest gap in that column is 1.4, between the mixed build and BF16, and even that is only t = +1.6. We publish ties as ties.

Two footnotes on the table. The BF16 row was served at 20,480 context against the quants' 32,768, because 74.89 GB of weights leaves only 81,664 tokens of KV on a 96 GB card; the generation budget was the same 16,384 tokens for every row. And out/answer is a mean over a long tail: median output is 208 tokens against a mean of 1,234, a ratio of 5.9×. That shape usually means a model that reaches its answer and cannot stop. Here it does not — the official FP8 shows 1,176 against a median of 211 on the same items, a ratio of 5.6×, with the same ~4% of items hitting the budget. It is how K2-Horizon reasons, and quantization did not cause it.

---

Throughput

Cache-free, 8,192 tokens in and 512 out, one RTX PRO 6000 Blackwell, same nightly and same serve flags as the accuracy rows. Prefix caching is disabled and every measurement runs on its own seed, so no row is reading a cache the previous one filled. These come from one sitting and drift a few percent between sittings, so read them as a bracket. BF16 is absent because at 74.89 GB it only serves 20,480 context on this card, which would not be the same measurement.

Quantizing this model buys throughput under load, not single-stream latency. Those three builds span 11.6 GB and land within 0.71 tok/s of each other at concurrency 1. At concurrency 32 their per-token decode cost is nearly flat too — 89.2, 89.5 and 93.1 ms, a 4.4% spread — so the 25% aggregate gain is not a decode-rate effect. It shows up in prefill and in scheduling under load, where median time to first token falls from 4.46 s to 1.54 s. Decode stays pinned because vLLM restacks the 15.1 GB of BF16 MoVA value-experts on every forward pass whatever the routed experts are quantized to, and that fixed cost dominates a single stream.

---

Choosing between our two builds

One axis separates our two builds: call accuracy, where the mixed build leads by 1.7 at t = +2.4. Its pooled tool-calling lead of 1.1 does not (t = +1.5), and neither does its knowledge (t = +0.9). So its extra 7.74 GB, +21%, buys better arguments on calls. This one is the faster and more portable artifact. If tool calling dominates your workload and you have 48 GB, IFM's own FP8 is still the most accurate option — it leads this build by 3.1 on call accuracy.

On Ampere. NVFP4 has no native kernel on sm_80; vLLM dequantizes through Marlin. Verified on an A100 40 GB: loads, serves and generates correctly with the reasoning parser working. That was a load-and-generate probe at 2,048 context, not a scored run — a 40 GB card leaves too little room for the 16,384-token protocol — so every number on this page comes from Blackwell. We have not tested the mixed build on Ampere and do not expect its FP8 block scales to load there.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above — the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, but it cannot tell you whether the model needed the room or failed to stop — finished and out/answer separate those, and this model's 95.6% is the lowest finished in the table because it reasons in long chains. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

The model is 37.4 B parameters in five very unequal pieces:

Which of those pieces got which format:

compressed-tensors, one config group. NVFP4 is W4A4, so the checkpoint carries observed activation scales alongside the weights as well as a tensor-level weight_global_scale.

The second-largest block cannot be quantized at all. vLLM's MoVA attention does this on every forward pass:

w1 = torch.stack([expert.weight for expert in self.v_experts], dim=0).contiguous()
v  = fused_mova_impl(..., w1=w1, ...)

It stacks the raw .weight of all 64 value-experts and hands fused_mova_impl a dense tensor. A quantized Linear carries weight_packed plus scales and no weight, so a quantized v_expert either fails at load or feeds packed bytes to a kernel that reads them as bfloat16. IFM's own FP8 ignores all 2,880 of them for the same reason. That is 20% of the model pinned to BF16 in any build, which is why 4 bits buys less here than the parameter count suggests: those 7.55 B parameters are 15.1 GB of BF16 that every forward pass moves regardless. Per the throughput section, that is why decode speed barely moves.

Attention is left alone too: MoVA fuses q and k into a single qk_proj, so those two would have to share one scheme and, under NVFP4, one weight_global_scale.

---

Attribution

• Base: IFM/K2-Horizon-MoVA-36B-A4B, Apache-2.0, BF16 weights unmodified.
• Built with llm-compressor 0.13.1 and compressed-tensors 0.18.1.
• Calibration: 496 conversations at seqlen 2048, used to observe activation scales.
• The checkpoint carries IFM's modeling_k2_horizon.py and configuration_k2_horizon.py unchanged, so --trust-remote-code works for tooling that wants the reference implementation.
• Audited before publication: every fused NVFP4 group shares one weight_global_scale, no activation scale is zero or NaN, and config.json agrees with the tensors actually present.

---

_{primitive · more models · inference economics for production LLM systems}