K2H traj-lr5e6 step 300
Continued pretraining of IFM/K2-Horizon-MoVA-36B-A4B toward LLM inference and ML systems engineering. This is step 300 of the recipe that kept general skills level with the release while the domain improved.
Research checkpoint, published 2026-11-04. It is not a release candidate.
Model
- Base: IFM/K2-Horizon-MoVA-36B-A4B (36B total, about 4B active, MoE). Same architecture, tokenizer and chat template.
- Weights: bf16 safetensors. Load with
trust_remote_code=True. - Sampling as the base model card recommends: temperature 1.0, top_p 0.95.
Training
- Trainer: xLLM, FSDP2, bf16, 3 GPUs, optimizer state offloaded to host memory.
- Start: the released weights.
- 300 steps. Sequence length 65,536, batch 1 per GPU, gradient accumulation 4: about 786k tokens per step, about 236M tokens in total.
- AdamW, beta1 0.9, beta2 0.95, weight decay 0.06, gradient clip 1.0.
- Learning rate 5e-6, 20 warmup steps, then constant.
Data mix (sampling weights):
- 44.5%: clean domain slices from Avifenesh/k2h-corpus-domain-clean (inference and kernel repositories, Linux, commit history) plus PR discussions and engineering receipts that are not published.
- 55.5%: script-verified replay as plain text from Avifenesh/k2h-replay-verified (
replay-synth-verified47.7%,replay-math-verified7.8%).
Evaluation
Paired per question against the release (same prompts, same sampling). McNemar p in brackets.
| Task | Release | Step 300 |
|---|---|---|
| IFEval prompt-strict (reasoning on, 2 runs pooled) | 83.7 | 82.0 (p=0.16) |
| GSM8K chat, reasoning on | 96.6 | 96.3 (p=0.52) |
| GSM8K 5-shot | 90.2 | 90.5 (p=0.76) |
| MMLU | 77.3 | 77.0 (p=0.51) |
| ARC-Challenge | 57.6 | 56.9 (p=0.28) |
| BoolQ | 88.4 | 88.3 (p=0.75) |
| MBPP pass@1 | 65.2 | 70.8 (p=0.001) |
| HumanEval pass@1 | 53.1 | 67.1 (p=0.0002) |
| Held-out PR NLL (lower is better) | 1.520 | 1.319 |
Passkey retrieval at 16k to 131k context: 5/5 at every length, as the release.
Higher learning rates (1.5e-5) gained a little more on the domain but broke the reply format: one arm ended turns with the document-end token, another stopped closing the reasoning block. At 5e-6 neither failure appeared.