K2-Horizon-4x0.9B-MoE-heretic
A 4-expert Mixture-of-Experts built from IFM's own K2-Horizon-0.9B checkpoints and uncensored with Heretic.
Usage · Results · How it was built · Limits · Credits
| Base | IFM/K2-Horizon-0.9B (Apache-2.0) |
| Architecture | native k2_horizon MoE (the model's own K2HorizonSparseMoeBlock) |
| Experts | 4 per layer, top-1 routing, softmax router |
| Parameters | 3.06B total, 1.08B active per token |
| Context | 128K (YaRN, same as the base) |
| Precision | bf16, 6.1 GB (GGUF quants) |
| Uncensoring | Heretic 2.0.0.dev0, Arbitrary-Rank Ablation |
Usage
Needs transformers 5.x and trust_remote_code=True (the architecture code ships with the repo).
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Lucasgitguy-co/K2-Horizon-4x0.9B-MoE-heretic"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", trust_remote_code=True, device_map="auto")
msgs = [{"role": "user", "content": "Explain how a transformer works in two sentences."}]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False, reasoning_effort="high")
ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=4096, do_sample=True, temperature=0.6, top_p=0.95)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
It is a reasoning model like the base: answers come after a <ifm|think> … </ifm|think> block. reasoning_effort accepts high, medium, low.
Results
| Model | Refusals (harmful_behaviors test, 100) | ARC-Easy acc / acc_norm (200, ±0.035) |
|---|---|---|
| IFM/K2-Horizon-0.9B | 100/100 | 0.560 / 0.575 |
| Dense 0.9B after Heretic (trial 47) | 4/100 (Heretic's own count: 0/100, KL 0.058) | – |
| This MoE | 3/100 | 0.570 / 0.560 |
Refusals: thinking skipped (</ifm|think> prefilled), 100 new tokens, greedy, Heretic's refusal keyword list. KL is Heretic's first-token KL divergence against the original on harmless prompts.
Routing share of held-out prompts per domain (top-1 picks, all layers):
| Prompts | main | rl_code2 | rl_if | rl_stem |
|---|---|---|---|---|
| general chat | 0.48 | 0.17 | 0.25 | 0.10 |
| code + math | 0.22 | 0.74 | 0.03 | 0.01 |
| instruction following | 0.21 | 0.13 | 0.63 | 0.04 |
| science | 0.35 | 0.22 | 0.09 | 0.35 |
The router was initialised from per-domain hidden-state means, so this table shows the routing works as set up, not that the experts are specialists (see Limits).
How it was built
- Experts. K2-Horizon-0.9B ships its RL specialist checkpoints as repo tags. Expert e of every layer is the MLP of one tag:
main(MOPD final, general),rl_code2(code and math),rl_if(instruction following),rl_stem(science). Attention, embeddings, norms andlm_headcome frommain. - Uncensoring. Heretic ran on dense
main(100 trials). Trial 47 changed 14 tensors:self_attn.o_projandmlp.down_projof layers 13–19. Theo_projedits were copied over as is; the samedown_projedit (W_heretic − W_main) was added to every expert'sdown_projin those layers. - Router. Per layer, gate row e = the normalised difference between the mean MLP input on domain-e prompts and the mean over all domains (computed on
main; 32 prompts per domain from harmless_alpaca, MBPP + GSM8K, IFEval, SciQ), scaled to unit logit spread. No router training. - Checks. A test build with every expert set to
main's MLP matched densemain's next-token argmax on 50/50 positions (max logit gap 0.72, bf16 noise).
# 1. sources (RL tags kept as MLP-only files to save disk)
hf download IFM/K2-Horizon-0.9B --revision main --local-dir src/main
for t in rl_code2 rl_if rl_stem; do
hf download IFM/K2-Horizon-0.9B --revision $t --local-dir src/tmp_$t \
--include "model*.safetensors" --include "model.safetensors.index.json"
python build/extract_mlp.py src/tmp_$t src/$t.mlp.safetensors && rm -rf src/tmp_$t
done
# 2. Heretic on dense main (config: build/config.toml), then save the chosen trial
heretic # 100 trials
python build/pareto.py checkpoints/src--main.jsonl
build/finish.sh 0 # saves trial, builds the MoE, runs checks
# 3. the MoE build on its own
python build/build_moe.py --src src --experts main rl_code2 rl_if rl_stem --top-k 1 \
--router hidden --heretic out/main-heretic --out out/K2-Horizon-4x0.9B-MoE-heretic
All scripts are in build/.
Limits
- Near-identical experts. The RL tags differ from
mainby only about 0.05% of their weights (relative norm;rl_ifvsrl_stemabout 0.07%). The MoE therefore behaves almost exactly like the base model, at 3× the size on disk. Real gains would need router or expert training, which was not done. - Runtimes. Transformers (this repo) and llama.cpp 0.9.0+ via the GGUF quants. MLX, Ollama and LM Studio were not tested.
- Uncensored. Refusals are largely removed. You are responsible for how you use the outputs.
- Tested on an Apple M4 (MPS, bf16) only.
Credits
- IFM for K2-Horizon-0.9B, its code and its released RL checkpoints (Apache-2.0).
- Heretic by Philipp Emanuel Weidmann for the abliteration.
- MoE background: the Hugging Face Mixture of Experts explained post.