K2H R1a final (step 800)
Continued pretraining of IFM/K2-Horizon-MoVA-36B-A4B toward LLM inference and ML systems engineering. Code and domain text improved; general skills, instruction following and tool calling stay level with the release.
Published 2026-11-07. This is the last checkpoint of the run that passes every gate below, tool calling included.
Model
- Base: IFM/K2-Horizon-MoVA-36B-A4B (36B total, about 4B active, MoE). Same architecture, tokenizer and chat template.
- Weights: bf16 safetensors. Load with
trust_remote_code=True. Context limit as the release (524,288). - Sampling as the base model card recommends: temperature 1.0, top_p 0.95.
Training
- Trainer: xLLM, FSDP2, bf16, 3 GPUs, optimizer state offloaded to host memory.
- Start: the released weights. 800 steps, sequence length 65,536, about 786k tokens per step, about 629M tokens.
- AdamW (0.9, 0.95), weight decay 0.06, gradient clip 1.0. Learning rate 5e-6, 20 warmup steps, then constant.
- Data mix (sampling weights):
- 40%: clean domain data: inference and kernel repositories, Linux, commit history (Avifenesh/k2h-corpus-domain-clean), plus PR discussions and engineering receipts that are not published.
- 50%: script-verified replay as plain text (Avifenesh/k2h-replay-verified).
- 10%: verified chat (reasoning, instruction and Q&A sets from Avifenesh/k2h-chat-reasoning-verified) used only as a release anchor: a hinged KL(release || model) on their reply tokens (tau 0.05), no cross-entropy. From step 600 the math replay also carries the anchor next to its cross-entropy.
- The anchor reads the release's top-64 log-probabilities from a precomputed cache, so no teacher runs during training.
Evaluation
Paired per question against the release, same prompts and sampling. A checkpoint passes when no task in the family below drops significantly after Holm correction, pooled IFEval (two runs) is not more than 2 points down at p < 0.05, chat GSM8K is not more than 1.5 points down, and MMLU is not 1 point or more down at p < 0.05.
| Task | Release | Step 800 | p |
|---|---|---|---|
| IFEval prompt-strict, reasoning on, 2 runs pooled | 83.7 | 84.7 | 0.47 |
| GSM8K chat, reasoning on | 96.6 | 96.2 | 0.41 |
| MATH-500 chat, reasoning on | 98.2 | 97.4 | 0.34 |
| MMLU | 77.3 | 76.7 | 0.18 |
| ARC-Challenge | 57.6 | 57.9 | 0.63 |
| BoolQ | 88.4 | 88.4 | 1.0 |
| MBPP pass@1 | 65.2 | 70.6 | 0.0009 |
| HumanEval pass@1 | 53.1 | 66.5 | 0.002 |
| BFCL V4 live_multiple | 75.0 | 72.5 | 0.03 |
| BFCL V4 irrelevance | 87.5 | 86.7 | 0.79 |
| BFCL V4 all live categories | 76.9 | 76.1 | 0.35 |
| Held-out PR NLL (lower is better) | 1.520 | 1.280 |
Long context (RULER, 13 tasks, 8 samples per task per length, paired with the release):
| Length | Release | Step 800 | p |
|---|---|---|---|
| 32k | 89.9 | 86.4 | 0.36 |
| 64k | 88.5 | 87.4 | 1.0 |
| 128k | 76.3 | 80.3 | 0.17 |
Measured again on 2026-11-08 with 24 samples per task (paired, 312 items per length): 32k 85.3 vs 90.4 (-5.1, p = 0.01), 64k 86.0 vs 87.2 (-1.2, p = 0.26). Long-context retrieval at 32k is a few points weaker than the release. Avifenesh/k2h-s1-final has no significant long-context drop and is the newer model.
Passkey retrieval at 16k to 131k context: 5/5 at every length, as the release.
Few-shot GSM8K (5-shot completion) drops 2.4 points (90.2 to 87.9). Chat GSM8K and MATH-500 hold, so this reads as drift in the few-shot completion format, not in math.
Step 1000 of the same run scores a little higher on the general tasks and on PR NLL (1.273), but BFCL live_multiple drops 3.6 points (p = 0.0015), so it is not the final checkpoint.