K2H c7-anchor step 300
Continued pretraining of IFM/K2-Horizon-MoVA-36B-A4B toward LLM inference and ML systems engineering, with a release anchor on chat data. This is the best checkpoint of the study so far: instruction following above the release, the same domain gain as the plain recipe.
Research checkpoint, published 2026-11-05. It is not a release candidate.
Model
- Base: IFM/K2-Horizon-MoVA-36B-A4B (36B total, about 4B active, MoE). Same architecture, tokenizer and chat template.
- Weights: bf16 safetensors. Load with
trust_remote_code=True. - Sampling as the base model card recommends: temperature 1.0, top_p 0.95.
Training
- Trainer: xLLM, FSDP2, bf16, 3 GPUs, optimizer state offloaded to host memory.
- Start: the released weights.
- 300 steps. Sequence length 65,536, batch 1 per GPU, gradient accumulation 4: about 786k tokens per step, about 236M tokens in total.
- AdamW, beta1 0.9, beta2 0.95, weight decay 0.06, gradient clip 1.0.
- Learning rate 5e-6, 20 warmup steps, then constant.
Data mix (sampling weights):
- 40%: clean domain slices from Avifenesh/k2h-corpus-domain-clean (inference and kernel repositories, Linux, commit history) plus PR discussions and engineering receipts that are not published.
- 50%: script-verified replay as plain text from Avifenesh/k2h-replay-verified.
- 10%: verified chat from Avifenesh/k2h-chat-reasoning-verified (
reasoning-ds-clean25%,instruct-clean3%,realchat-clean2%), trained with no cross-entropy. These tokens only carry the anchor.
Release anchor: on the chat tokens, the loss adds a hinged KL to the frozen release, sum max(0, KL(release || model) - tau), with tau 0.05 and coefficient 1. The KL uses the release top-64 tokens plus one tail bucket. The model keeps the release's own chat behaviour and never trains on those answers.
Evaluation
Paired per question against the release (same prompts, same sampling). McNemar p in brackets. "Plain recipe" is the same run without the anchor, published as Avifenesh/k2h-traj-lr5e6-s300.
| Task | Release | Plain recipe | This model |
|---|---|---|---|
| IFEval prompt-strict (reasoning on, 2 runs pooled) | 83.7 | 82.0 | 85.3 (p=0.18) |
| GSM8K chat, reasoning on | 96.6 | 96.3 | 96.4 (p=0.82) |
| GSM8K 5-shot | 90.2 | 90.5 | 89.6 (p=0.49) |
| MMLU | 77.3 | 77.0 | 76.4 (p=0.03) |
| ARC-Challenge | 57.6 | 56.9 | 57.7 (p=1.0) |
| BoolQ | 88.4 | 88.3 | 87.9 (p=0.10) |
| MBPP pass@1 | 65.2 | 70.8 | 71.4 (p=0.0004) |
| HumanEval pass@1 | 53.1 | 67.1 | 66.5 (p=0.0003) |
| Held-out PR NLL (lower is better) | 1.520 | 1.319 | 1.322 |
- Against the plain recipe, IFEval is 3.3 points higher (paired, p about 0.005) with the same domain gain.
- MMLU is 0.9 points under the release. That is the cost of the anchor at this length.
- Passkey retrieval at 16k to 131k context: 5/5 at every length, as the release.
- Mean KL to the release on held-out chat text is about half that of the plain recipe (instruct 0.006 vs 0.011, real chat 0.018 vs 0.041, reasoning 0.005 vs 0.014). On math, synthetic and PR text it is unchanged.