baseline-nanochat-d24
The stock nanochat architecture
(@ 92d63d4) trained at the d24 (~1.38 B param) scale. This is the baseline
in a controlled comparison against a two‑pass routing‑loop variant
(kz919/looped-nanochat-d24).
Code: https://github.com/kyleliang919/looped_nanochat
Results (vs. looped, identical training budget)
| Metric | Baseline | Looped |
|---|---|---|
| Base CORE | 0.2552 | 0.2783 |
| ChatCORE (full sets) | 0.2253 | 0.2521 |
| ARC‑Easy | 62.04% | 68.01% |
| ARC‑Challenge | 50.00% | 52.22% |
| MMLU | 37.06% | 37.96% |
| GSM8K | 2.88% | 4.17% |
| HumanEval | 10.98% | 10.98% |
The looped model wins or ties every metric at ~2.4× the training compute. See the full report.
Model details
| Layers | 24 |
n_embd |
1536 |
| Heads / KV heads | 12 / 12 |
| Vocab | 32768 |
| Context | 2048 |
| Window pattern | SSSL |
| Routing loop | off (standard nanochat) |
| Params | ~1.38 B |
| Precision | bf16 (fp8 matmuls in training) |
Training: pretrained for 5,568 steps on ~5.84 B tokens
(--target-param-data-ratio=8, 1,048,576‑token batches), then chat‑SFT on
SmolTalk + MMLU(×3) + GSM8K(×4). This checkpoint is the SFT model (step 467).
Files
model_000467.pt— model weights (state_dict)meta_000467.json—model_config+ training configtokenizer.pkl,token_bytes.pt— nanochat tokenizer
Usage
Uses the nanochat runtime (not 🤗 Transformers). Place the files into
~/.cache/nanochat/chatsft_checkpoints/baseline-d24/ and
~/.cache/nanochat/tokenizer/, then:
python -m scripts.chat_cli -g baseline-d24 -i sft -p "Why is the sky blue?"
Limitations
A ~$100‑tier micro‑model: it hallucinates and is weak at math/code. Research artifact, not for production.
License
MIT (inherited from nanochat).