K2H main run, step 900
Step 900 of a planned 7,391-step continued-pretraining run on IFM/K2-Horizon-MoVA-36B-A4B. The run was stopped here because general skills were drifting down; the shorter clean-data arms (k2h-c1-clean, k2h-c2-sft, k2h-c4-ctrl) followed.
Research checkpoint, published 2026-11-03. It is not a release candidate. Compare it with the release, not in isolation.
Model
- Base: IFM/K2-Horizon-MoVA-36B-A4B (36B total, about 4B active, MoE). Same architecture, tokenizer and chat template.
- Weights: bf16 safetensors. The custom modeling code ships with the weights, so load with
trust_remote_code=True. - Sampling as the base model card recommends: temperature 1.0, top_p 0.95.
Training
- Trainer: xLLM, FSDP2, bf16, 3 GPUs.
- Start: the released weights of IFM/K2-Horizon-MoVA-36B-A4B.
- 900 steps of a planned 7,391. Sequence length 32,768, about 786k tokens per step, about 0.71B tokens in total.
- AdamW, beta1 0.9, beta2 0.95, weight decay 0.06, gradient clip 1.0.
- Learning rate 1.5e-5, 100 warmup steps, warmup-stable-decay schedule (stopped in the stable phase).
Data mix (sampling weights):
- Domain code, commits, pull requests and papers before the verification pass, plus synthetic and web replay, a math slice and a small TxT360 instruct share.
- The cleaned code and commit slices are in Avifenesh/k2h-corpus-domain-clean. The web replay is in Avifenesh/k2h-sources-unverified.
Evaluation
All numbers come from the same harness as the release column (lm-evaluation-harness against a vLLM server). IFEval and chat GSM8K use the chat template with reasoning on, temperature 1.0, top_p 0.95, 14,336 new tokens, and score only the text after the reasoning block. The PR NLL is the mean token NLL on 150 held-out pull requests (530k tokens).
| Task | Release | This model | Change |
|---|---|---|---|
| GSM8K, 5-shot, strict | 90.1 | 84.6 | -5.5 |
| GSM8K, chat zero-shot (reasoning on) | 96.6 | not measured | |
| IFEval, prompt-strict (reasoning on) | 84.3 | 71.2 | -13.1 |
| MMLU | 77.3 | 76.0 | -1.3 |
| ARC-Challenge | 57.6 | 56.4 | -1.2 |
| ARC-Easy | 85.6 | 85.3 | -0.3 |
| BoolQ | 88.4 | 87.5 | -0.9 |
| HellaSwag (norm) | 67.6 | 67.5 | -0.1 |
| PIQA | 80.5 | 81.0 | +0.5 |
| WinoGrande | 69.1 | 70.6 | +1.5 |
| MBPP pass@1 | 65.2 | 69.6 | +4.4 |
| HumanEval pass@1 (4 samples) | 53.0 | 63.4 | +10.4 |
| Held-out PR NLL (lower is better) | 1.519 | 1.173 | -0.346 |
What the numbers say:
- The held-out PR NLL drops from 1.519 to 1.173: the domain gain is large.
- General skills drift down: GSM8K -5.5, IFEval -13.1, MMLU -1.3, ARC-Challenge -1.2. Coding goes up (MBPP +4.4, HumanEval +10.4).
- Chat-format GSM8K was not measured for this checkpoint.
- The full training state (with optimizer) is in Avifenesh/k2h-ckpt900-xllm.
License
Apache-2.0, the license of the base model.