One recipe, five sizes: TinyBrainBot v4 scaling study (500K to 25M)
Five language models from 0.48M to 25.2M non-embedding parameters, trained on exactly the same 11.0B tokens in exactly the same order with the same schedule and tokenizer. Only the model size (and its width-scaled peak learning rate) changes, so the differences between them are the effect of size alone.
Results
EleutherAI lm-eval-harness v0.4.13, 0-shot, up to 2,000 examples per task. Each task's first row
is acc_norm where the task reports it (acc otherwise); the "(raw acc)" row under it is acc.
The average uses the first rows. Pythia was run by us on the same harness and settings.
| Benchmark | 25M | 10M | 5M | 2M | 500K | Pythia-70M | Pythia-31M |
|---|---|---|---|---|---|---|---|
| ARC-Easy | 42.7 | 40.2 | 37.8 | 34.1 | 32.8 | 36.0 | 33.7 |
| ARC-Easy (raw acc) | 47.3 | 42.8 | 40.4 | 35.2 | 31.4 | 38.2 | 37.2 |
| ARC-Challenge | 24.9 | 23.1 | 21.5 | 21.2 | 21.1 | 21.8 | 21.0 |
| ARC-Challenge (raw acc) | 21.0 | 18.9 | 17.2 | 16.5 | 18.2 | 17.6 | 16.6 |
| SciQ | 65.9 | 58.4 | 59.7 | 51.7 | 48.5 | 56.4 | 51.5 |
| SciQ (raw acc) | 73.7 | 68.8 | 65.6 | 57.2 | 47.9 | 64.0 | 60.0 |
| OpenBookQA | 31.2 | 28.2 | 27.2 | 27.0 | 24.2 | 25.4 | 26.8 |
| OpenBookQA (raw acc) | 17.2 | 15.4 | 14.0 | 14.2 | 11.8 | 12.6 | 13.0 |
| MMLU | 23.7 | 23.1 | 23.0 | 23.1 | 23.2 | 22.9 | 22.9 |
| HellaSwag | 37.2 | 35.1 | 34.6 | 34.9 | 31.8 | 33.8 | 33.6 |
| HellaSwag (raw acc) | 32.5 | 32.2 | 31.9 | 31.1 | 30.0 | 30.8 | 31.2 |
| PIQA | 60.2 | 57.3 | 56.5 | 55.9 | 54.0 | 59.2 | 56.6 |
| PIQA (raw acc) | 60.0 | 59.6 | 57.6 | 57.7 | 56.0 | 59.8 | 57.5 |
| WinoGrande | 52.6 | 51.5 | 49.0 | 48.9 | 52.0 | 52.9 | 49.3 |
| Social IQa | 37.2 | 36.1 | 35.6 | 35.5 | 34.3 | 35.4 | 35.1 |
| CommonsenseQA | 20.3 | 20.6 | 19.9 | 19.6 | 19.2 | 19.8 | 19.7 |
| LAMBADA (OpenAI) | 23.4 | 17.6 | 14.6 | 10.4 | 5.5 | 23.4 | 17.8 |
| BoolQ | 58.8 | 60.8 | 59.1 | 50.2 | 37.2 | 59.6 | 51.0 |
| MathQA | 21.9 | 21.9 | 22.3 | 21.2 | 19.8 | 20.4 | 19.5 |
| MathQA (raw acc) | 23.2 | 22.0 | 22.2 | 21.6 | 20.2 | 20.9 | 19.2 |
| Average (13 tasks) | 38.5 | 36.5 | 35.4 | 33.4 | 31.0 | 35.9 | 33.7 |
| LAMBADA perplexity (lower is better) | 125 | 311 | 584 | 1451 | 4894 | 129 | 270 |
Non-embedding parameters: 25.18M, 9.99M, 5.12M, 1.97M, 0.48M; Pythia-70M 18.9M, Pythia-31M 4.7M.
What it shows
- A clean, log-linear curve. The 13-task average rises by roughly 1.3 points per doubling of non-embedding parameters, steadily from 500K to 25M: 31.0, 33.4, 35.4, 36.5, 38.5.
- About half the parameters for the same average as Pythia. The 10M (10.0M non-embedding) averages above Pythia-70M (18.9M), and the 2M (2.0M) is within 0.3 of Pythia-31M (4.7M), having trained on 11B tokens against Pythia's 300B.
- LAMBADA is the exception, and it favours Pythia. LAMBADA perplexity falls as a near power law with size (4,894 to 125), but at matched size Pythia is clearly better: Pythia-31M (270) beats this 5M (584) and even the 10M (311). Long-range narrative prediction is what 300B tokens of the Pile buy; a broader, more factual 11B-token mix buys the other benchmarks.
- Steepest size effects: LAMBADA, SciQ, ARC-Easy and BoolQ. Flattest: MMLU, CommonsenseQA, MathQA and ARC-Challenge, which sit at or near chance for every model here.
- The 500K scores 37.2 on BoolQ, below a coin flip. That is what answering "no" almost every time looks like, since roughly 62% of BoolQ answers are "yes". By 2M it is gone.
The models
| Folder | Non-embedding | Embedding | Total | Embedding share | Shape (width x layers) | Peak LR | Training time |
|---|---|---|---|---|---|---|---|
25m |
25.18M | 5.38M | 30.55M | 18% | 384 x 16, 6 heads of 64, 2 KV, FFN 1024 | 2e-3 | ~22 h, 2 GPUs |
10m |
9.99M | 3.58M | 13.57M | 26% | 256 x 14, 4 heads of 64, 2 KV, FFN 672 | 3e-3 | ~21 h |
5m |
5.12M | 2.69M | 7.80M | 34% | 192 x 13, 3 heads of 64, 1 KV, FFN 512 | 4e-3 | ~17 h |
2m |
1.97M | 1.79M | 3.76M | 48% | 128 x 10, 2 heads of 64, 1 KV, FFN 384 | 6e-3 | ~11 h |
500k |
0.48M | 0.90M | 1.38M | 65% | 64 x 9, 1 head of 64, 1 KV, FFN 192 | 1e-2 | ~8 h |
At the bottom of the range the 14k-token embedding is most of the model: 65% of the 500K's parameters. Embeddings are tied (input and output share one matrix).
Identical training
- Tokenizer: one 14,000-token byte-level BPE (digits split one per token, newlines kept, no unknown tokens) shared by all five.
- Data: the same packed, block-shuffled corpus files, read in the same order, so every model
sees the same 131,072-token batch at every step.
- Stage 1, 9.7B tokens: web 37% (FineWeb-Edu, DCLM), reference 13.5% (Wikipedia), narrative 11% (Project Gutenberg, BookCorpusOpen), code 9%, math 9% (MegaMath, FineMath), synthetic textbooks 7.5% (Cosmopedia), Q&A and facts 7%, educational PDFs 5.5%, reasoning 0.5%.
- Stage 2, 500M tokens: the same mix plus 7.8% generated everyday-commonsense, procedural, worked-arithmetic and short-fact text written for this purpose. No benchmark data.
- Stage 3, 803M-token anneal: a fact-dense mix while the learning rate decays to zero.
- Schedule: warmup-stable-decay. Stage 1 is scheduled to 76,697 steps with an 8% decay and stops at step 74,000, early in that decay; stage 2 re-warms over 300 steps and runs to 83,931 steps with the final 7.3% decaying over the anneal.
- Optimizer: AdamW (0.9, 0.95), weight decay 0.1, gradient clip 1.0, warmup 1,000 steps, fp16 mixed precision, context 1,024.
- Hardware: 2x Tesla V100 16GB. The 25M ran across both GPUs; the smaller models ran one per GPU, two at a time, with gradient accumulation matching the same batch.
What varies: size, and peak learning rate scaled inversely with width (2e-3 at width 384). The 500K is capped at 1e-2 rather than the rule's 1.2e-2.
Limitations
- One seed per size. Gaps under about 1 point on a single task are not resolvable here.
- Learning rates follow a rule, not a sweep. A per-size sweep could move individual points.
- The Pythia comparison crosses everything: data, tokenizer, token count and training code all differ. It places the curve, it does not isolate a cause.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, size = "nkthebass/tinybrainbot-v4-scaling-500k-25m", "10m" # 25m, 10m, 5m, 2m or 500k
tok = AutoTokenizer.from_pretrained(repo, subfolder=size)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=size)
ids = tok("Once upon a time,", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60)[0], skip_special_tokens=True))
These are base models: they continue text. The 25M is also released on its own as TinyBrainBot-25M-v4-Base.
Reproduce the benchmarks
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-v4-scaling-500k-25m,subfolder=10m,dtype=float32 \
--tasks arc_easy,arc_challenge,sciq,openbookqa,mmlu,hellaswag,piqa,winogrande,social_iqa,commonsense_qa,lambada_openai,boolq,mathqa \
--num_fewshot 0 --limit 2000 --batch_size 8 --seed 1234
Raw lm-eval output for every model, Pythia included, is in results/, with exact parameter counts
in results/param_counts.json.
Related
- Do data-mix rankings transfer across scale?: the companion study that holds size fixed at 100M and varies the data.