TinyBrainBot-10M-Base
Superseded by TinyBrainBot-10M-v2-Base: a general-purpose 10M (Int 13.42) trained from scratch on 10B tokens.
A 9.9M-parameter language model trained from scratch on ~10B tokens, as a plain Llama (LlamaForCausalLM), so it runs in transformers, llama.cpp, LM Studio and Jan with no custom code.
Scores vs other models of 10M parameters or fewer
Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other rows come from that leaderboard.
| Model | Params | Int Index | HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| TinyBrainBot-10M-Base (this model) | 9.90M | 17.61 | 28.28 | 36.66 | 24.23 | 58.81 | 65.40 |
| (raw acc) | 17.24 | 27.44 | 38.30 | 20.31 | 59.41 | 65.50 | |
| Pascaline-3M (ReckonerLabs) * | 2.98M | 16.09 | 27.79 | 34.18 | 22.44 | 56.15 | 69.20 |
| ForgePlex-M2.2-10M | 9.99M | 12.12 | 29.19 | 39.77 | 22.70 | 59.79 | 37.40 |
| heron1i-flash-lite-v2 | 9.92M | 11.11 | 28.86 | 38.89 | 23.38 | 58.54 | 36.70 |
| min-spark-1.6 | 9.96M | 11.08 | 28.52 | 38.30 | 22.44 | 59.14 | 36.90 |
* Pascaline-3M's figures are from its model card (it was not yet on the leaderboard at the time of writing). We reproduced them with the same tools.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-10m-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-10m-base")
ids = tok("Mia has 9 fewer pens than Leo. Leo has 30 pens. Mia has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=8, do_sample=False)[0]))
GGUF: tinybrainbot-10m-base-Q8_0.gguf (11 MB) and -F16.gguf (20 MB), for llama.cpp, LM Studio and Jan.
This is a base model: it continues text and does not chat. At this size, expect simple prose, grade-school arithmetic stories, and plenty of made-up facts.
Model
| Parameters | 9,902,336 total (8.85M non-embedding + 1.05M tied embedding) |
| Shape | hidden 256, 12 layers, 4 query heads of 64 with 2 KV heads (GQA), SwiGLU 704, RMSNorm, RoPE |
| Vocabulary | 4,096-token byte-level BPE: digits split one per token, whitespace kept exactly, no unknown tokens. Numbers are written normally (no custom number tokens) |
| Context | 1,024 tokens |
Training
~9.95B tokens: a constant learning-rate phase of 8.9B tokens, then a 1.04B-token decay. Hardware: 2× Tesla V100.
| Constant-LR phase | Share |
|---|---|
| Q&A and facts (our distilled plain-text sets) | 22% |
| Cosmopedia Stanford + OpenStax | 21% |
| Grade-school story arithmetic (generated by our own programs) | 20% |
| TinyStories | 15% |
| FineWeb-Edu | 10% |
| Everyday / how-to (Cosmopedia WikiHow + distilled) | 7% |
| Wikipedia leads | 5% |
| Decay phase (1.04B tokens) | Share |
|---|---|
| Generated grade-school arithmetic, several phrasing families | 45% |
| The pretraining sets above, re-weighted toward Q&A, facts and commonsense | 40% |
| Cosmopedia WikiHow | 15% |
The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions of a whole, patterns, tallies, and especially relations ("Mia has 9 fewer pens than Leo"). Relations are asked in both directions, with the unknown on either side, and as minimal pairs that flip only "more" and "fewer". No benchmark training or test data was used.
Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 3e-3 with a warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision.
How this checkpoint was chosen. From the constant-LR checkpoint at step 68,000, we ran five decay branches with different arithmetic data. They scored between 15.20 and 17.61 on the Index. This is the best one, and the published weights are that branch's final checkpoint, not a merge. The branches that included relational arithmetic scored 16.4–17.6.
Evaluation
Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-10m-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-10m-base --dtype float32
Limitations
- English only, and very small: it confabulates freely, so don't rely on it for facts.
- Its arithmetic is grade-school sized and works best in familiar story phrasings. Free-form answers are less reliable than its ranked-choice benchmark scores suggest.
- Base model only: no instruction tuning and no safety tuning.