TinyBrainBot-3M-v2-Base

A 2.94M-parameter general-purpose base model trained from scratch on 6.05B tokens, as a plain Llama (LlamaForCausalLM): no custom code. It is the successor of TinyBrainBot-3M-Base, rebuilt to be balanced: commonsense, facts and Q&A first, with grade-school arithmetic that holds up on wordings it never saw, instead of a high ArithMark score.

Scores vs other models of 3M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row shows plain accuracy. The Intelligence Index is the Open SLM Leaderboard's; the other models' numbers come from that leaderboard.

Arithmetic it never trained on

The arithmetic share was tuned so ArithMark-3 stays close to how the model does on problems written by other people. None of these sets was used in training (Gemma and Llama were never used to write training data), all acc_norm:

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-3m-v2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-3m-v2-base")
ids = tok("To keep a cut apple from turning brown, you can", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=20, do_sample=False)[0]))

There is also an F16 GGUF (tinybrainbot-3m-v2-base-F16.gguf, 6 MB) for llama.cpp, LM Studio and Jan. A Q8_0 file isn't possible because the hidden size (144) is not a multiple of 32.

This is a base model: it continues text and does not chat. At this size expect short, simple text.

Model

Training

6.05B tokens in three parts: the first 4.72B are TinyBrainBot-3M-Base's own pretraining (its step 36,000), then 0.81B of new pretraining and a 0.51B decay.

The arithmetic mixes several sources so the skill is not tied to one wording: drills generated by our own programs (order of operations, negatives, number formats, number words, whitespace variants, tables, unknown positions, distractors; every answer computed in Python), story problems in sentence frames written by several LLM families (Ling, Mistral; the numbers filled in and computed by Python), and Orca-Math word problems. No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), LR 2.7e-3 held then decayed to 0 (warmup-stable-decay), 131k tokens per step, fp16 mixed precision, 2x V100.

How this checkpoint was chosen. Several decay branches with different arithmetic shares were run from the same checkpoint (step 42,212). A little story-style arithmetic raised ArithMark-3 much faster than any held-out set, so we published the branch whose public ArithMark-3 stays within about 10 points of the held-out sets, not the one with the highest Index (13.93, but ArithMark-3 62.8 against 41.0 held-out). The published weights are that branch's final checkpoint, not a merge.

Evaluation

Float32, lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-v2-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-v2-base --dtype float32

Limitations

At 3M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic is unreliable outside simple one-step problems. HellaSwag and ARC are close to chance for every model this size, so the gaps between models there are small.