CLOD-149M (v1)
A 148.9M-parameter decoder-only language model trained from scratch on a single RTX 5050 8 GB Laptop GPU in 8 days. 20B tokens: ~18.2B of ClimbMix-400B (single epoch, no repeats) followed by a ~10% terminal annealing phase of procedurally generated arithmetic word problems (55%) mixed with held-out ClimbMix text (45%), scheduled to coincide with LR warmdown.
Specs
| Parameters | 148,898,078 (sub-150M class) |
| Architecture | 10 layers, d=512, 4 heads, full attention, RoPE, value embeddings |
| Optimizer | Muon (matrices) + AdamW (embeddings/head), nanochat defaults |
| Tokenizer | custom 32,768-vocab BPE (rustbpe), single-digit number splitting |
| Tokens | 20.0B (415:1 tokens-per-scaling-param) |
| Batch | 524,288 tokens/step, seq len 2048 |
| Final val_bpb | 0.831965 |
Evaluation
| Task | Score |
|---|---|
| HellaSwag | 34.86 |
| ARC-Easy / ARC-Challenge | 45.37 / 26.88 |
| PIQA | 67.08 |
| ArithMark-3 | 36.00 |
| Intelligence Index (Open SLM formula) | 19.64 |
Known limitations
- Confidently believes Michelangelo painted the Mona Lisa and that quartz is the hardest natural substance - majority-vote errors absorbed from web text.
- Arithmetic is completion-format trained; multiple-choice transfer is partial (ArithMark 36%). v2 will anneal in MC format.
- Greedy samples loop; use temperature ≥ 0.8.