TinyBrainBot-100M-v5.1-Base
A 100M-parameter base model trained from scratch on 29.6B tokens, as a plain Llama (LlamaForCausalLM): no custom
code. It is TinyBrainBot-100M-v3-Base continued for 4.6B
tokens with knowledge-dense data, and it beats it on every Open SLM Leaderboard task, without any benchmark training data.
Scores
Zero-shot, full test sets, acc_norm; (raw acc) underneath. Intelligence Index from the Open SLM Leaderboard.
| Model | Params | Int Index | HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| TinyBrainBot-100M-v5.1-Base (this model) | 100.1M | 26.07 | 34.98 | 57.15 | 34.47 | 67.52 | 47.00 |
| (raw acc) | 24.87 | 30.83 | 62.92 | 30.29 | 67.57 | 47.00 | |
| TinyBrainBot-100M-v3-Base (previous) | 100.1M | 20.97 | 32.62 | 54.67 | 30.12 | 66.21 | 37.40 |
| void.1 | 90.2M | 24.38 | 39.5 | 49.3 | 29.4 | 67.2 | 43.7 |
| MobileLLM-R1-140M-base | 140M | 24.64 | 33.8 | 49.9 | 24.7 | 63.2 | 65.7 |
| BananaMind-2-Pro | 139M | 24.96 | 42.8 | 53.6 | 27.8 | 67.5 | 38.2 |
| GPT-X2.5-135M | 135M | 25.17 | 40.6 | 51.8 | 29.2 | 69.4 | 38.4 |
Other models' numbers are from the leaderboard.
Arithmetic it never trained on
ArithMark-3 is kept honest: on 1,380 held-out arithmetic problems in topics and wordings the model never saw, it scores 40.7, close to its public 47.0.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-100m-v5.1-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-100m-v5.1-base")
ids = tok("The water cycle begins when the sun heats water in oceans and lakes, causing it to", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=24, do_sample=False)[0]))
GGUF files for llama.cpp, LM Studio, Ollama and Jan: F16 (201 MB), Q8_0 (107 MB) and Q4_K_M (66 MB), with
add_space_prefix=false set so they tokenize exactly like transformers (greedy output matches).
This is a base model: it continues text and does not chat.
Model
| Parameters | 100.1M (75.5M non-embedding + 24.6M tied embedding) |
| Shape | hidden 768, 12 layers, 12 query heads of 64 with 4 KV heads (GQA), SwiGLU 2048, RMSNorm, RoPE |
| Vocabulary | 32,000 SentencePiece |
| Context | 1,024 tokens |
Training
29.6B tokens: the 25.06B of 100M-v3-Base, then 4.56B tokens in five stages, each re-warming the learning rate (peak 3e-4 to 6e-4) and decaying it again. The published weights are the final step of the last stage (step 113,000), not a merge.
| Continuation data (4.56B tokens) | Approx. share |
|---|---|
| Q&A and facts: our distilled sets, GooAQ, FactBase-DCLM, finephrase FAQ, ELI5 | 30% |
| Web: DCLM-Ultra-1.1, ClimbMix, top-scored FineWeb-Edu pages (score 4-5) | 23% |
| Textbooks: Cosmopedia (incl. SmolLM cosmopedia-v2, school-level audiences), WikiHow | 21% |
| Wikipedia (dense and Simple English) | 11% |
| Commonsense, how-to and writing (our distilled sets, finephrase tutorials) | 6.5% |
| Arithmetic (our drills, Orca-Math, UltraData-Math) | 4.5% |
| Grade-school science Q&A (final stage only) | 3% |
| Narrative (Gutenberg, BookCorpus) | 1% |
Grade-school science Q&A. The final stage includes about 420k grade-school science questions with short explained answers, written by Ling-3.0-Flash: no answer options, and any item sharing an 8-word phrase with an ARC question was removed before training.
No benchmark train or test data was used in the continuation.
Evaluation
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-100m-v5.1-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-100m-v5.1-base --dtype float32
Limitations
A 100M base model: good at short factual and commonsense continuations for its size, but open-ended generation is limited and can be wrong. Not aligned or safety-tuned. Its chat version is TinyBrainBot-100M-v5.1-Instruct.