TinyBrainBot-3M-Base
Superseded by TinyBrainBot-3M-v2-Base: a balanced general 3M (Int 10.38) whose arithmetic holds up on held-out wordings.
A 2.94M-parameter language model trained from scratch on 5.55B tokens, as a plain Llama (LlamaForCausalLM): no custom code.
Scores vs other models of 3M parameters or fewer
Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other models' numbers come from that leaderboard.
| Model | Params | Int Index | HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| TinyBrainBot-3M-Base (this model) | 2.94M | 11.33 | 26.94 | 33.12 | 23.04 | 55.98 | 51.20 |
| (raw acc) | 11.55 | 26.63 | 34.72 | 18.94 | 57.73 | 50.50 | |
| GPT-S3-3M | 2.97M | 9.61 | 27.68 | 34.76 | 23.12 | 57.02 | 39.10 |
| Tokle-3M | 2.91M | 8.91 | 27.20 | 34.85 | 23.98 | 55.01 | 40.80 |
| pulvis-v2 | 2.96M | 8.62 | 27.93 | 31.86 | 22.78 | 56.86 | 37.40 |
| Purrence-3M | 2.99M | 8.49 | 27.81 | 34.55 | 23.63 | 56.64 | 34.80 |
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-3m-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-3m-base")
ids = tok("Ava had 23 apples. Ava got 19 more. Now Ava has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=6, do_sample=False)[0]))
# -> ... 42 apples
There is also an F16 GGUF (tinybrainbot-3m-base-F16.gguf, 6 MB) for llama.cpp, LM Studio and Jan. A Q8_0 file isn't possible because the hidden size (144) is not a multiple of 32.
This is a base model: it continues text and does not chat. At this size, expect short, simple text and grade-school arithmetic in familiar phrasings.
Model
| Parameters | 2,942,928 total (2.35M non-embedding + 0.59M tied embedding) |
| Shape | hidden 144, 10 layers, 3 query heads of 48 with 1 KV head (GQA), SwiGLU 416, RMSNorm, RoPE |
| Vocabulary | 4,096-token byte-level BPE trained for this model: digits split one per token, whitespace kept exactly, no unknown tokens |
| Context | 1,024 tokens |
Training
5.55B tokens: 4.72B of pretraining, then a 0.83B-token decay.
| Pretraining | Share |
|---|---|
| Q&A and facts (our distilled plain-text sets) | 25% |
| Grade-school story arithmetic (generated by our own programs) | 20% |
| Cosmopedia Stanford + OpenStax | 15% |
| TinyStories | 15% |
| Everyday / how-to (Cosmopedia WikiHow + distilled) | 10% |
| FineWeb-Edu | 10% |
| Wikipedia leads | 5% |
| Decay (0.83B tokens) | Share |
|---|---|
| The pretraining sets, re-weighted toward Q&A, facts and commonsense | 50% |
| Generated grade-school arithmetic in many phrasings | 35% |
| Cosmopedia WikiHow | 15% |
The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions, patterns and tallies, each in many different sentence shapes. No benchmark training or test data was used.
Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 4e-3 with a warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision, trained on 2× V100.
How this checkpoint was chosen. We ran four decay branches with different arithmetic data from the checkpoint at step 36,000. They scored between 8.80 and 11.33 on the Index. This is the best one, and the published weights are that branch's final checkpoint, not a merge.
One thing we learned. Our first version learned its arithmetic from a single sentence pattern. In that pattern it solved 2-digit sums with carrying 99% of the time, but in other wordings it was near chance, and it scored 33.6 on ArithMark-3. At this size, the skill gets tied to the wording, and spreading the same arithmetic over many phrasings is what fixed it.
Evaluation
Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-base --dtype float32
Limitations
At 3M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic only works reliably in common phrasings. HellaSwag and ARC are close to chance for every model this size, so the gaps between models there are small.