TinyBrainBot-360M-A75M-v5.1-Base
A mixture-of-experts base model trained from scratch on 17.9B tokens: 383M parameters in total, 75M active per
token (24 experts per layer, 4 used per token). It loads as a standard MixtralForCausalLM (no custom code). It
replaces TinyBrainBot-360M-A75M-v4.1-MoE-Base,
with a new final training stage built around school-level knowledge, and it beats it on every Open SLM Leaderboard task.
Scores
Zero-shot, full test sets, acc_norm; (raw acc) underneath. Intelligence Index from the Open SLM Leaderboard.
| Model | Params | Int Index | HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 |
|---|---|---|---|---|---|---|---|
| TinyBrainBot-360M-A75M-v5.1-Base (this model) | 383M (75M active) | 29.76 | 38.68 | 58.96 | 36.18 | 68.01 | 53.00 |
| (raw acc) | 27.93 | 32.75 | 64.60 | 33.36 | 67.68 | 53.00 | |
| TinyBrainBot-360M-A75M-v4.1-MoE-Base (previous) | 383M (75M active) | 24.40 | 37.27 | 55.64 | 32.34 | 66.32 | 42.00 |
| GPT-X3-150M | 149M | 33.05 | 46.2 | 59.6 | 31.3 | 71.3 | 50.9 |
| Boris-2-Preview-2 | 144M | 30.58 | 46.6 | 59.1 | 30.2 | 70.8 | 42.2 |
| KLM-1.5 | 99M | 30.26 | 43.7 | 59.7 | 32.7 | 68.5 | 48.2 |
| Sol Pro | 138M | 28.60 | 46.7 | 54.0 | 31.8 | 71.0 | 36.0 |
Other models' numbers are from the leaderboard.
Arithmetic it never trained on
On 1,380 held-out arithmetic problems in topics and wordings the model never saw, it scores 47.3, close to its public ArithMark-3 of 53.0.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-360m-a75m-v5.1-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-360m-a75m-v5.1-base")
ids = tok("The water cycle begins when the sun heats water in oceans and lakes, causing it to", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=30, do_sample=False, repetition_penalty=1.0)[0]))
GGUF files for llama.cpp, LM Studio, Ollama and Jan: F16 (768 MB), Q8_0 (409 MB) and Q4_K_M (239 MB). Greedy output matches transformers token for token.
This is a base model: it continues text and does not chat. Its chat version, TinyBrainBot-360M-A75M-v5.1-Instruct, is in the TinyBrainBot v5.1 collection.
Model
| Parameters | 383.4M total; 75.7M active non-embedding per token + 24.6M tied embedding |
| Shape | hidden 768, 12 layers, 12 query heads of 64 with 4 KV heads (GQA), RMSNorm, RoPE |
| Experts | 24 SwiGLU experts of width 512 per layer, top-4 routing |
| Vocabulary | 32,000 SentencePiece; line breaks are kept in this version (earlier TinyBrainBot models on this tokenizer never saw them) |
| Context | 2,048 tokens |
Training
17.9B tokens: 16.7B of pretraining at constant learning rate, then a new 1.2B-token decay.
| Pretraining (16.7B) | Share |
|---|---|
| Educational web: FineWeb-Edu, DCLM, finepdfs-edu | 44% |
| Wikipedia | 13% |
| Books and stories: Gutenberg, BookCorpus, TinyStories | 11% |
| Math: MegaMath, FineMath, Orca-Math | 9% |
| Code: the-stack-smol | 9% |
| Cosmopedia (Stanford, OpenStax, math) | 7% |
| Q&A, facts and reasoning (our distilled sets) | 7% |
| Decay (1.2B) | Share |
|---|---|
| Q&A and facts: our distilled sets, GooAQ, FactBase-DCLM, finephrase FAQ, ELI5 | 26% |
| School-level textbooks: SmolLM cosmopedia-v2 and Cosmopedia, school audiences | 22% |
| Educational and broad web: top-scored FineWeb-Edu pages, DCLM-Ultra-1.1, ClimbMix, WikiHow | 22% |
| Wikipedia (dense and Simple English) | 12% |
| Grade-school science Q&A | 10% |
| Commonsense, how-to and writing (our distilled sets) | 5% |
| Arithmetic (our drills, Orca-Math, UltraData-Math) | 3% |
The grade-school science Q&A (about 420k questions with short explained answers, no answer options) was written by Ling-3.0-Flash; any item sharing an 8-word phrase with an ARC question was removed before training. No benchmark train or test data was used.
Recipe: AdamW, peak LR 8e-4 with a warmup-stable-decay schedule, 262k tokens per step, fp16, 2x V100.
Evaluation
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-360m-a75m-v5.1-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-360m-a75m-v5.1-base --dtype float32
Limitations
A small base model: good at short factual and commonsense continuations for its size, but open-ended generation is limited and can be wrong. Not aligned or safety-tuned.