TinyBrainBot-25M-v5-Base
A 30.6M-parameter language model (25.2M non-embedding) trained from scratch on ~12B tokens, as a plain Llama (LlamaForCausalLM): no custom code, so it runs in transformers, llama.cpp, LM Studio and Jan.
Scores
Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other models' numbers come from that leaderboard.
| Model | Params | Int Index | HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| TinyBrainBot-25M-v5-Base (this model) | 30.6M | 24.61 | 30.29 | 41.46 | 26.54 | 63.17 | 76.30 |
| (raw acc) | 23.11 | 28.53 | 45.12 | 21.42 | 62.08 | 76.30 | |
| void.1 | 90.1M | 23.92 | 38.68 | 47.31 | 28.16 | 67.46 | 44.80 |
| 100M-exp | 98.2M | 22.47 | 37.78 | 49.83 | 27.22 | 66.97 | 40.00 |
| 50M-pro | 49.6M | 17.93 | 33.25 | 45.62 | 25.51 | 64.80 | 37.40 |
| gibc-v2-lm | 49.8M | 16.13 | 31.87 | 47.85 | 26.11 | 60.28 | 40.20 |
| TinyBrainBot-25M-v4-Base (the starting point) | 30.6M | 12.54 | 28.86 | 42.09 | 24.91 | 60.23 | 35.20 |
It is the highest-scoring model we know of at 100M parameters or fewer, at about a third of the size of the next one.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-25m-v5-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-25m-v5-base")
ids = tok("Mia has 9 fewer pens than Leo. Leo has 30 pens. Mia has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=6, do_sample=False)[0]))
# -> ... Mia has 21 pens.
GGUF: tinybrainbot-25m-v5-base-Q8_0.gguf (33 MB) and -F16.gguf (62 MB).
This is a base model: it continues text and does not chat. Expect fluent simple prose, grade-school arithmetic stories, and plenty of made-up facts.
Model
| Parameters | 30,554,496 total (25.18M non-embedding + 5.38M tied embedding) |
| Shape | hidden 384, 16 layers, 6 query heads of 64 with 2 KV heads (GQA), SwiGLU 1024, RMSNorm, RoPE |
| Vocabulary | 14,000-token byte-level BPE: digits split one per token, whitespace kept exactly, no unknown tokens |
| Context | 1,024 tokens |
Training
This model continues TinyBrainBot-25M-v4-Base, which saw ~11B tokens of general pretraining: web (FineWeb-Edu, DCLM), Wikipedia, books (Project Gutenberg, BookCorpus), Cosmopedia, math web pages, educational PDFs, code, and our distilled Q&A and everyday-explanation sets.
v5 adds one more 1.04B-token decay phase. The learning rate re-warms to 2e-3 over 300 steps, then decays to zero on this mix:
| Decay phase | Share |
|---|---|
| Generated grade-school arithmetic, several phrasing families | 45% |
| Q&A, facts, commonsense, Cosmopedia, Wikipedia, TinyStories | 40% |
| Cosmopedia WikiHow | 15% |
The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions of a whole, patterns, tallies, and especially relations ("Mia has 9 fewer pens than Leo"). Relations are asked in both directions and as minimal pairs that flip only "more" and "fewer". No benchmark training or test data was used.
This was a single run, not a selection among branches. The published weights are its final checkpoint, not a merge.
Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), 131k tokens per step, fp16 mixed precision, 2× Tesla V100.
Evaluation
Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-25m-v5-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-25m-v5-base --dtype float32
Limitations
- English only, and small: it confabulates freely, so don't rely on it for facts.
- Its arithmetic is grade-school sized and works best in familiar story phrasings. Free-form answers are less reliable than its ranked-choice benchmark scores suggest.
- Base model only: no instruction tuning and no safety tuning.