K2-Horizon-MoVA-36B-A4B FAST2 🚀

3.1× faster decode at 61K context. 2.1× faster prefill. Weights bit-identical to APEX-Mini. Zero perplexity loss inside the window.

This is a recipe repo — it deliberately contains no model weights. FAST2 is not a different file: it is the original VladHong/K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf plus a 160-byte metadata key (an 8192-token sliding-window attention declaration) and the companion llama.cpp fork that honors it. Build it yourself in under a minute, below.

The numbers (measured, single session, 61,337-token prompt, 5 interleaved rounds)

At short context FAST2 matches vanilla (~59 tok/s — weight-bandwidth bound, untouched by design). Raw aggregates and spreads (0.4–2.1%): results-head2head.json.

* Vanilla with its default f16 KV cache cannot serve 61K context on a 22 GB GPU at all (OOM); the comparator is vanilla with q8_0 KV — the only configuration in which vanilla fits.

Get it running (two steps)

1. Get the base GGUF — VladHong/K2-Horizon-MoVA-36B-A4B-APEX-Mini-GGUF

2. Get the patched llama.cpp — VladHong/llama-cpp-K2-FAST2 (complete source snapshot; the same code is branch k2-fast2 on ModelScope). Build it exactly like the MBZUAI fork, plus -DGGML_CUDA_FA_ALL_QUANTS=ON:

cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_CURL=OFF \
      -DCMAKE_CUDA_ARCHITECTURES=<your-sm> -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

Then either (a) window at load time — no file changes at all:

llama-server -m K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf -c 65536 -ngl 99 --parallel 1 \
    -fa on -ctk q4_0 -ctv f16 -b 2048 -ub 2048 -t 8 \
    --override-kv k2-horizon.attention.sliding_window=int:8192 --port 8099

or (b) bake the window into a self-contained GGUF (~12 seconds, from this repo's make_swa_gguf.py):

python3 make_swa_gguf.py K2-Horizon-MoVA-36B-A4B-APEX-Mini.gguf 8192 K2-FAST2-SWA8K.gguf
llama-server -m K2-FAST2-SWA8K.gguf -c 65536 -ngl 99 --parallel 1 -fa on -b 2048 -ub 2048 -t 8 --port 8099

fast2-serve.sh wraps all of this with VRAM gating and context-aware flags (FAST2_SWA=8192 to enable the window through the launcher).

Quality: measured, not hand-waved

• Perplexity inside the window is numerically identical to full attention (2.1814 vs 2.1814 on a 240K-token stratified corpus; Δ 0.0006 at 16K context with W=16384).
• 61K-context needle retrieval: PASS inside the window — exact answer; outside it the model declines instead of hallucinating (clean failure mode).
• The tradeoff is explicit: attention recall covers the last W tokens (8192 here; rebuild with W=16384 → 43.90 tok/s @61K, still 2.7× vanilla, recall 16K).

Why it's fast (one paragraph)

K2-Horizon decodes slowly at long context because all 48 layers run full attention over the entire history (~192 KiB of KV per token — 6+ GiB read per generated token at 61K). The window caps each layer's KV, so decode cost flattens instead of growing, and the f16 KV path (measured 23% faster than q8_0 on this kernel despite 2× the bytes) becomes affordable at 61K. We also measured and rejected the "obvious" tricks: KV quantization below f16 is slower here (dequant cost > bytes saved), and bigger micro-batches only help prefill.

Files in this repo

Honesty clause

Every claim comes from gated, interleaved, repeated measurement published in results-head2head.json — the same discipline that showed the previous "FAST" build of this model was not faster at all. FAST2 changes no weights (tensors byte-identical to APEX-Mini, sha-verified), does not touch short-context speed, and removes attention recall beyond the window by design. Provenance: base weights IFM K2-Horizon-MoVA-36B-A4B (apache-2.0); APEX-Mini quant by the reference repo; measured on RTX 2080 Ti 22 GB.