Byrne-15M-Looped
Small toy / smoke-test for Byrne-100M-Ultra-MC. I ran looping here first, at ~15M, to see if the stack still trained before I spent the 114M budget. Not a product. Not a matched scale-up of Ultra-MC.
~15.2M active / ~18.9M total looped SpikeWhale (MoE, loop_count=3,
effective depth 30). Trained from scratch on UltraX, then SFT on
UltraChat, then DPO. Inference code, checkpoints, and a safetensors
export for leaderboard owners are in this repo.
It's a small model. Fluent-ish English, weak knowledge, weak code. Numbers in Evaluation. Base continues text. SFT/DPO take ChatML; they are a probe, not a chat product.
The scored pretrained dir is safetensors/base/ (WikiText-2 byte_ppl
2.943). Pretrain tokens: 246M at the 40k matched A/B,
419M at step 68.25k. SFT adds 82M packed tokens (10k × batch 8
× 1024). Almost all of the budget is the base.
Checkpoints
| file | stage | use |
|---|---|---|
safetensors/base/ |
pretrained (checkpoints/best) |
scored. Lowest byte_ppl. Continues text. |
checkpoints/base_40k.pt |
pretrain step 40k | loop vs unlooped A/B. FINDINGS.md |
checkpoints/base_68k.pt |
pretrain step 68.25k | later base, val_ppl 59.8 |
checkpoints/sft_ultrachat10k.pt |
SFT 10k UltraChat | --chat |
checkpoints/dpo_seg1.pt |
DPO on the SFT | --chat (default). Scored. |
Same loader for all of them. Context is 1024 everywhere. Safetensors
mirrors are under safetensors/ — that is what to point a harness at.
Architecture
Looped decoder (model_v2.py, SpikeWhale v2). MoE is on. Vocab 16,512
via a byte-level Length-MAX tokenizer (spike_tokenizer.py,
tokenizer.json) from Dong & Su, Length-MAX Tokenizer for Language
Models (arXiv:2511.20849).
| Active / total params | 15.2M / 18.9M |
| Hidden size | 384 |
| Layers | 10 × loop_count 3 (effective 30) |
| Attention | 8 heads, 1 KV (MQA), MLA |
| Head dim | 64 (RoPE 32 + NoPE 32) |
| FFN | MoE, 4 routed + 1 shared, top-2, width 160 |
| Tie embeddings | yes |
| Context | 1024 |
Also on: fractal RoPE (γ=1), Elo, Engram (table 1024), HRM refine ×3 with deep supervision, abstain head, XSA, hyper-connections. No Memory Cache. That branch lives on Ultra-MC.
Config: byrne_15m.yaml + the config baked into each .pt.
The +1,152 params vs the unlooped 15M baseline are per-pass embeddings. Everything else is weight-tied across the three passes.
Evaluation
Full, uncapped. Same harness as Ultra-MC. Raw JSON:
benchmarks/Byrne-15M-Looped.json
(base) and
benchmarks/Byrne-15M-DPOseg1.json.
Tables: BENCHMARKS.md.
| Metric | base | DPO seg1 | chance |
|---|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.943 | 3.014 | — |
| BLiMP acc ↑ | 0.734 | 0.711 | 0.50 |
| arc_easy acc | 0.335 | 0.325 | 0.25 |
| arc_easy acc_norm | 0.328 | 0.326 | 0.25 |
| arc_challenge acc | 0.182 | 0.179 | 0.25 |
| arc_challenge acc_norm | 0.224 | 0.232 | 0.25 |
| hellaswag acc | 0.267 | 0.269 | 0.25 |
| hellaswag acc_norm | 0.272 | 0.271 | 0.25 |
| winogrande acc | 0.509 | 0.512 | 0.50 |
| piqa acc | 0.533 | 0.554 | 0.50 |
| piqa acc_norm | 0.521 | 0.539 | 0.50 |
| openbookqa acc | 0.126 | 0.138 | 0.25 |
| openbookqa acc_norm | 0.248 | 0.262 | 0.25 |
| boolq acc | 0.378 | 0.397 | 0.50 |
| ArithMark-3.0 acc | 0.345 | 0.322 | 0.25 |
| ArithMark-3.0 acc_norm | 0.347 | 0.327 | 0.25 |
n: arc_easy 2376, arc_challenge 1172, hellaswag 10042, winogrande 1267, piqa 1838, openbookqa 500, boolq 3270, arithmark 1000, BLiMP 1800. Not a 200-cap.
Base wins LM metrics and ArithMark. DPO moves PIQA / BoolQ a little and costs PPL. Most MC is chance. 15M.
Loop vs no-loop (step 40k, 246M tokens)
Param-matched A/B against loop_count=1. Looped spends 3× compute per
token. Writeup: FINDINGS.md. Checkpoint: base_40k.pt.
Val loss 4.195 vs 4.231. Sampled repetition 0.011 vs 0.049. The
unlooped baseline's % % % collapse (rep 0.92) never showed up on
the looped arm.
68k, capped 200
checkpoints/base_68k.pt. Same harness, --limit 200. Does not
replace the full table above. JSON:
benchmarks/Byrne-15M-Looped-base68k-cap200.json.
| Metric | 68k cap-200 | chance |
|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.937 | — |
| BLiMP acc ↑ | 0.728 | 0.50 |
| arc_easy acc | 0.360 | 0.25 |
| hellaswag acc_norm | 0.330 | 0.25 |
| piqa acc | 0.515 | 0.50 |
| boolq acc | 0.365 | 0.50 |
| ArithMark-3.0 acc_norm | 0.335 | 0.25 |
n=200 on MC / ArithMark. Full table: BENCHMARKS.md.
Usage
# base (text continuation) — scored pretrained dir
python generate.py --ckpt safetensors/base -p "The capital of France is" --temp 0.7 --top-k 40 --rp 1.3
# DPO chat (default checkpoint)
python generate.py --chat -p "Explain why the sky is blue in one sentence." --temp 0.7 --top-k 40 --rp 1.3
# SFT chat
python generate.py --ckpt checkpoints/sft_ultrachat10k.pt --chat -p "Give me a tip for staying focused."
Defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy is a test tool.
DECODING-DEFAULTS.md.
python verify.py # greedy cached decode == full recompute
python safetensors/load.py base
Files
config.py model_v2.py spike_tokenizer.py special_tokens.py fractal.py
chat_format.py generate.py tokenizer.json
byrne_15m.yaml package.json requirements.txt
verify.py DECODING-DEFAULTS.md PROVENANCE.md FINDINGS.md
checkpoints/{base_40k,base_68k,sft_ultrachat10k,dpo_seg1}.pt
safetensors/{base,base_40k,base_68k,sft_ultrachat10k,dpo_seg1}/
benchmarks/
Leaderboard: safetensors/base/ (config.json + model.safetensors +
tokenizer.json). You still need this repo's model_v2.py. Call
tie_weights() after load. safetensors/README.md.
Training recipe
Pretrain: UltraX-Preview (UltraX-Ultra-FineWeb), batch 6, seq 1024,
lr 6e-4, cosine 500k planned. Released bases: 40k (246M tokens) and
68.25k (419M). SFT: UltraChat train_sft, 10k steps, batch 8, block
1024, from checkpoints/best. DPO: UltraFeedback, 1k steps, β=0.1,
seg1 is the scored chat stage.
Limitations
- 15M. Factual recall is weak. MC tasks near chance.
- Code and multi-step reasoning are weak.
- English. No safety tuning past DPO.
- Base continues text. Chat template will not make it answer questions.
- Looping costs 3× FLOPs per token vs the dense 15M baseline.
What this is
A toy. Smoke-test for Byrne-100M-Ultra-MC. I wanted to know the looped stack would train at all before I put 2.0B tokens into the 114M run.
While I was here I also checked whether loop_count=3 beats
loop_count=1 at equal parameters. It does, on val loss and on
degeneration, at 3× the compute per token. Then I SFT'd and DPO'd it
the same way I do the bigger Byrne runs. That is extra. The point of
this repo is the smoke test.
SFT/DPO are a probe. If you want instruct, start from safetensors/base
and train your own.
Citation
@misc{byrne15mlooped,
title = {Byrne-15M-Looped: A ~15M-parameter looped SpikeWhaleLM},
author = {Dean Byrne (Quazim0t0)},
year = {2026},
howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-15M-Looped}},
note = {Quazim0t0/Byrne-15M-Looped}
}
Updated 2026-09-20.