K2-Horizon-4x0.9B-MoE-heretic-GGUF
GGUF quants of K2-Horizon-4x0.9B-MoE-heretic for llama.cpp.
Files · Usage · How they were made · Limits · Credits
| Source | Lucasgitguy-co/K2-Horizon-4x0.9B-MoE-heretic (bf16, Apache-2.0) |
| Architecture | k2-horizon in llama.cpp, 4 experts per layer, top-1 routing, softmax router |
| Parameters | 3.06B total, 1.08B active per token |
| Context | 128K (YaRN) |
| Chat template | embedded; works with --jinja, reasoning is split into reasoning_content |
| Runtime | llama.cpp 0.9.0 (build 12110) or newer |
Files
| File | Size | Perplexity | KLD vs BF16 | Same top token | Notes |
|---|---|---|---|---|---|
K2-Horizon-4x0.9B-MoE-heretic-BF16.gguf |
6.12 GB | 5.371 | 0 | 100% | reference, lossless |
K2-Horizon-4x0.9B-MoE-heretic-Q8_0.gguf |
3.25 GB | 5.377 (+0.1%) | 0.0010 | 98.7% | near-lossless |
K2-Horizon-4x0.9B-MoE-heretic-Q6_K.gguf |
2.51 GB | 5.399 (+0.6%) | 0.0099 | 95.7% | very close to BF16 |
K2-Horizon-4x0.9B-MoE-heretic-Q5_K_M.gguf |
2.18 GB | 5.434 (+1.2%) | 0.0320 | 92.6% | recommended |
K2-Horizon-4x0.9B-MoE-heretic-Q4_K_M.gguf |
1.87 GB | 5.745 (+7.0%) | 0.1189 | 87.0% | noticeable loss |
K2-Horizon-4x0.9B-MoE-heretic-Q4_K_S.gguf |
1.76 GB | 5.906 (+10.0%) | 0.1381 | 85.9% | noticeable loss |
K2-Horizon-4x0.9B-MoE-heretic-Q3_K_M.gguf |
1.49 GB | 6.763 (+26.0%) | 0.2998 | 79.2% | heavy loss, only if memory is very short |
Measured with llama-perplexity --kl-divergence on 40 × 512 tokens of English technical text (llama.cpp's README and docs), each quant downloaded from this repo. With only 1.08B active parameters the model is sensitive to quantization: Q5_K_M is the smallest file that stays close to BF16. All quants are plain llama.cpp quants without an importance matrix. Q2_K was made too but removed: perplexity 28.9 (5.4× BF16) and the same top token only 50% of the time.
Usage
# chat in the terminal
llama-cli -hf Lucasgitguy-co/K2-Horizon-4x0.9B-MoE-heretic-GGUF:Q5_K_M --jinja
# OpenAI-compatible server on :8080
llama-server -hf Lucasgitguy-co/K2-Horizon-4x0.9B-MoE-heretic-GGUF:Q5_K_M --jinja -c 8192
It is a reasoning model like its base: it thinks inside <ifm|think> … </ifm|think> before answering. With --jinja, llama-server returns the thinking as reasoning_content and the answer as content. The base model's suggested sampling is --temp 0.6 --top-p 0.95.
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"What is 17*3? Answer briefly."}],"temperature":0}'
On the BF16 file this returned reasoning_content "We need to compute 17*3 = 51. Provide answer briefly." and content "51".
How they were made
convert_hf_to_gguf.pyfrom llama.cpp (commit a4d4f2f, 2026-11-09) turned the bf16 safetensors intoBF16.gguf. The converter readsrouter_score_func: softmaxand writes it as the expert gating function.llama-quantizefrom llama.cpp 0.9.0 made every other file fromBF16.gguf.- Each file was checked with a short greedy completion before upload, then downloaded again from this repo for the size check and KL-divergence measurement. The BF16 file was also checked in
llama-server --jinjafor chat-template rendering and reasoning parsing.
python llama.cpp/convert_hf_to_gguf.py K2-Horizon-4x0.9B-MoE-heretic \
--outtype bf16 --outfile K2-Horizon-4x0.9B-MoE-heretic-BF16.gguf
for Q in Q8_0 Q6_K Q5_K_M Q4_K_M Q4_K_S Q3_K_M; do
llama-quantize K2-Horizon-4x0.9B-MoE-heretic-BF16.gguf K2-Horizon-4x0.9B-MoE-heretic-$Q.gguf $Q
done
Limits
- Near-identical experts. The four experts come from IFM's RL checkpoints, which differ from
mainby about 0.05% of their weights. The model behaves almost exactly like the dense 0.9B base, at 3× the size. See the source card. - New architecture. Older llama.cpp builds fail with
unknown model architecture: 'k2-horizon'. Ollama, LM Studio and other llama.cpp-based apps work only once they ship a llama.cpp withk2-horizonsupport; none of them were tested. - Uncensored. Refusals are largely removed (3/100 on Heretic's test set for the bf16 model). You are responsible for how you use the outputs.
- Refusal and ARC-Easy numbers are for the bf16 model only; the quants were measured with perplexity/KLD only. Tested on an Apple M4 (Metal) only.