K2-Horizon-MoVA-36B-A4B-DERISKED GGUF

TQ1_0 and TQ2_0 do not produce readable text. Every other quant that was generation-tested does. Files with "not tested" in the table have been uploaded and not checked.

GGUF conversions of Blackfrost-AI/K2-Horizon-MoVA-36B-A4B-DERISKED-BF16. The source is a weight-level behavioral derivative of IFM/K2-Horizon-MoVA-36B-A4B. DERISKED is the Blackfrost research label. It is not a claim of safety, zero refusals, or unchanged benchmark scores.

Upstream llama.cpp cannot load k2-horizon. Build the IFM fork at 42adf01:

https://github.com/MBZUAI-IFM/llama.cpp/tree/model/K2Horizon

Generation tests

Same prompt every time: The capital of France is, temperature 0, --jinja. Host: Ryzen 6850H / Radeon 680M. Fork build b1-42adf01. GGML_VK_DISABLE_ASYNC=1 on the Vulkan run.

A readable sample (IQ4_XS):

[Start thinking]
1.  **Analyze the User's Request:**
    *   The user

Q5_K_M opened the same way: Analyze the User's Request: The user's query is ". The ternary files did not. They repeated loose words with no sentence.

CPU speeds above are not a benchmark. Several of those runs shared the machine with quantization. IQ3_M is the quant that actually ran on the 680M GPU. Q8_0, Q6_K, and Q5_K_M refused -ngl 99 with failed to fit params to free device memory.

Routers (attn_v_gate) stay F32. No imatrix was used. The imatrix pass exited during load.

Q4_K_DYN keeps the output tensor and attention projections at Q6_K, token embeddings at Q5_K, and routers at F32. Everything else is Q4_K_M. This fork has no Unsloth Dynamic quant type.

Run

git clone --branch model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp
cd llama.cpp
git checkout 42adf01
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-cli

GGML_VK_DISABLE_ASYNC=1 ./build/bin/llama-server \
  -m K2-Horizon-MoVA-36B-A4B-DERISKED-IQ3_M.gguf \
  -ngl 99 -c 4096 -fa on --jinja \
  --temp 1.0 --top-p 0.95 \
  --host 0.0.0.0 --port 8082

The chat template is custom. llama.cpp applies it only with --jinja. Upstream sampling is temperature 1.0, top_p 0.95, reasoning effort high. Thinking is wrapped in <ifm|think>.

GGML_VK_DISABLE_ASYNC=1 matters on the 680M. Async Vulkan uploads have produced ? / NaN garbage on this GPU for other models.

If Windows prints Failed to process regex, the MSVC regex engine is rejecting the tokenizer pattern. Use the same fork on Linux.

Architecture

All 36B weights have to sit in RAM or VRAM. "4B active" is the compute per token, not the file size.

MTP head and oxidize-c (262K context)

Short version: the MTP head works, but it does not make decoding faster. On the test box, plain decode was faster in every case measured. It is published for research and for anyone who wants to improve the verify path. For speed, use the plain IQ3_M file with oxidize-c and compressed KV (commands below).

Files

The head is a single DeepSeek-V3-style "nextn" layer (one dense K2 block plus eh_proj / enorm / hnorm / shared_head_norm). It reuses the target's token_embd and output. It was trained on hidden states from this IQ3_M target; mtp/SPEC.md has the details.

The merged file contains the 798 IQ3_M tensors byte-for-byte unchanged, followed by the 14 head tensors copied byte-for-byte from the sidecar. The only changed metadata key is block_count (48 → 49); the nextn_* keys were added.

Runtime support

• The head runs only in oxidize-c, the C11 engine on branch feat/oxidize-c-k2-horizon of the oxidize repository (github.com/Zapdev-labs/oxidize). When this was written, that branch had not yet been pushed publicly.
• The llama.cpp K2 fork (model/K2Horizon) and stock llama.cpp cannot use the MTP layer. For llama.cpp, use the plain K2-Horizon-MoVA-36B-A4B-DERISKED-IQ3_M.gguf, not the -MTP file.
• In oxidize-c, MTP is opt-in with --mtp (greedy only, default 1 draft token). Without --mtp, the merged file decodes exactly like the plain IQ3_M file, with the head left unused.

make -C oxidize-c            # AVX2 CPU build
# merged file
./oxidize-c/oxidize-c --model K2-Horizon-MoVA-36B-A4B-DERISKED-IQ3_M-MTP.gguf --threads 8 --mtp --prompt "..."
# or: plain IQ3_M + sidecar
./oxidize-c/oxidize-c --model K2-Horizon-MoVA-36B-A4B-DERISKED-IQ3_M.gguf \
  --mtp-model mtp/mtp-K2-Horizon-MoVA-36B-A4B-DERISKED-Q8_0.gguf --threads 8 --mtp --prompt "..."

Measured MTP results

Host: Ryzen 7 6850H (8 cores / 16 threads, AVX2), 23 GB RAM, CPU only, 8 threads, --ctx 262144 with RQ-compressed KV, greedy, 128 generated tokens. Speed is relative to plain oxidize-c decode on the same prompt.

• CLI run at --ctx 262144: 7.6–7.8 tok/s with --mtp (26% accepted), against 11.2–11.6 tok/s without it.
• Why MTP loses: one verify step costs about 1.7× a single decode step (about 150 ms against 88 ms). At the acceptance rates above, that never pays back.
• Offline eval (mtp/train_eval.json): top-1 agreement with the target of 38.7%, and an expected accepted length of 0.54 tokens at 2 draft tokens.
• Output: the MTP output matches plain decode for the first 32–45 tokens and can then drift. Batched and single-row matmuls round differently, so the drift is expected.

Recommended: oxidize-c at 262K context, no MTP

./oxidize-c/oxidize-c --model K2-Horizon-MoVA-36B-A4B-DERISKED-IQ3_M.gguf \
  --ctx 262144 --threads 8 --prompt "..."

At --ctx 262144, oxidize-c automatically picks RQ-compressed KV: 3-bit keys and values, an exact window over the most recent 1024 positions, and 4 sink tokens. --kv rq selects it explicitly. The KV cache is allocated lazily and costs about 38 KB per filled token.

Decode speed in tok/s, same host, 8 threads:

• Depth 0: oxidize-c and llama.cpp are level.
• 4K, real prompt: oxidize-c is about 1.2× faster than llama.cpp's best setting.
• 32K: RQ is about 1.6× faster than q8 KV. Decode still slows a lot as the context fills.
• Prefill in the 262K CLI run was 14.7 tok/s.
• Memory:
  • The 262K CLI launch peaked at 15.9 GB RSS with no swap. Nearly all of that is the memory-mapped weights; anonymous memory was 118 MB.
  • A completely full 262K RQ cache would need about 10 GB on top of about 16.5 GB of weights. A 23 GB machine can therefore fill roughly 130K tokens before weights start being evicted.
• Shorter contexts: below --ctx 8192 the default KV type is f32, which is the slowest option. Pass --kv q8: at a real 4K depth it gave 9.2 tok/s, against 7.4 tok/s for f32.

License

Apache-2.0, same as the source card. See LICENSE. Architecture, tokenizer, and chat template remain IFM's. The checkpoint weights are the Blackfrost derivative. This repository only repackages those weights as GGUF.

Lineage

IFM/K2-Horizon-MoVA-36B-A4B@7730b92d1b574e04663b04023d5d6fa83475432f
└── Blackfrost-AI/K2-Horizon-MoVA-36B-A4B-DERISKED-BF16
    └── freakyskittle/K2-Horizon-MoVA-36B-A4B-DERISKED-GGUF