Consecutive-token tiny llama (14.2M params)
Causal LM trained from scratch on consecutive token-id runs
[k, k+1, ...] mod 50000 (ids 0..49999; 49999 wraps to 0). Greedy
decoding continues the run exactly, which makes it a hand-checkable oracle
for verifying inference runtimes (vLLM, etc.) token-for-token against the
HuggingFace reference.
- Architecture:
llama(random init, trained 3000 steps, lr 0.0005, batch 16, boundary-crossing frac 0.25) - Data: fully synthetic, generated on the fly (
train.py::consecutive_seq) - Eval (greedy exact-match of the hand-computed continuation):
in-distribution
1.000, longer-than-train1.000