tinybrainbot-10m-v2-base (ONNX)

This is an ONNX version of nkthebass/tinybrainbot-10m-v2-base. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js

See the pipeline documentation for text-generation: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextGenerationPipeline

---

TinyBrainBot-10M-v2-Base

A 9.9M-parameter general-purpose base model trained from scratch on 10.16B tokens, as a plain Llama (LlamaForCausalLM): no custom code. It succeeds TinyBrainBot-10M-Base and is built like TinyBrainBot-3M-v2-Base: web text, Q&A, facts and commonsense first, with arithmetic as a small part of the mix.

Scores vs other models of 10M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row shows plain accuracy. The Intelligence Index is the Open SLM Leaderboard's; the other models' numbers come from that leaderboard.

More arithmetic results

Arithmetic sets the model never trained on (Gemma and Llama were never used to write training data), all acc_norm:

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-10m-v2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-10m-v2-base")
ids = tok("The best way to keep bread fresh is to", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=24, do_sample=False)[0]))

GGUF files for llama.cpp, LM Studio and Jan: tinybrainbot-10m-v2-base-F16.gguf (20 MB) and tinybrainbot-10m-v2-base-Q8_0.gguf (11 MB).

This is a base model: it continues text and does not chat. At this size expect short, simple text.

Model

Training

10.16B tokens: 8.57B of pretraining at constant learning rate, then a 1.59B-token decay.

The arithmetic comes from several sources so the skill is not tied to one wording: drills generated by our own programs (number formats, number words, whitespace variants, negatives, order of operations, tables, unknown positions, distractors; every answer computed in Python), Orca-Math and UltraData-Math word problems, and a very small amount of story problems in LLM-written sentence frames. No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 3e-3 with 1,000 warmup steps, warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision, trained on 2x V100 in ~8 hours.

How this checkpoint was chosen. Several decay branches with different arithmetic shares were run from the same checkpoint (step 65,303); this is one of them, and the published weights are that branch's final checkpoint, not a merge.

Evaluation

Float32, lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-10m-v2-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-10m-v2-base --dtype float32

Limitations

At 10M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic is unreliable beyond simple problems. HellaSwag and ARC are close to chance for every model this size.