TinyBrainBot-360M-A75M-v4.1-MoE-Base

A mixture-of-experts language model trained from scratch on two Tesla V100s. Every layer holds 24 experts and each token uses 4 of them, so the model stores 359M parameters but runs only 75.7M per token (non-embedding; plus a 24.6M tied embedding, 383M in total). It was trained on 18.2B tokens with a 2,048-token context, then given one expert-tuned patch for factual question answering (below).

It is our strongest model on the Open SLM Leaderboard's settings, ahead of our dense 350M while computing about a quarter as much per token.

Full test sets, zero-shot, acc_norm, EleutherAI lm-eval-harness, plus the official ArithMark-3 script. The Intelligence Index is the leaderboard's: the normalised average of HellaSwag, ARC, PIQA and 0.65x ArithMark-3. TinyBrainBot rows were run by us; the other rows are the leaderboard's published numbers.

Quick start

It is a standard Mixtral model and needs no custom code:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Once upon a time, a little fox", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80)       # defaults from generation_config.json
print(tok.decode(out[0], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Ollama): tinybrainbot-360m-a75m-v4.1-moe-base-Q8_0.gguf (409 MB) or -F16.gguf (768 MB). Both tokenize exactly like the transformers version and give the same greedy output.

llama-completion -m tinybrainbot-360m-a75m-v4.1-moe-base-Q8_0.gguf -no-cnv -p "Once upon a time, a little fox" -n 80 --temp 0.7 --repeat-penalty 1.15

This is a base model: it continues text, it does not follow chat instructions. Default sampling: temperature 0.7, top_p 0.9, repetition_penalty 1.15. A chat version is in progress.

Samples

Unedited, at the default settings.

Once upon a time, a little fox named Max lived in a small village. One day, Max heard his name mentioned by someone else. He asked around the town and found that everyone had lost their way in the forest. But then, he noticed something strange happening - some trees were open wide! All of the animals were moving quickly toward each other because they wanted to find food and shelter. The fox realized that the path wasn't clear at all; instead, it seemed like it

Question: Why do we have seasons on Earth? Answer: The tilt of Earth's axis causes changing sunlight angles.

The best way to learn a new language is by engaging with it. This means watching movies, reading books, attending cultural events, and practicing speaking the language regularly. It's not just about memorizing words; it's also about understanding grammar rules and vocabulary lists. By doing so, you will gain confidence in communicating effectively and expressing yourself in the target language. Another important aspect of learning a new language is improving your listening skills. Listening to native speakers can help you understand conversations and

All benchmarks

lm-eval-harness, 0-shot, up to 2,000 examples per task (HellaSwag and MMLU use their first 2,000; the leaderboard table above uses full sets). Each task's first row is acc_norm where the task reports it, acc otherwise; the "(raw acc)" row under it is acc. "Before the expert patch" is the same model at the end of pretraining.

Model

Routing is Mixtral's: softmax over the 24 experts, keep the top 4, renormalise their weights. A 0.01 load-balancing loss keeps every expert in use. The experts run on the sigma-MoE Triton kernel (Csordas et al.), which fuses token routing into the matrix multiply, so training ran at about 30k tokens/s on the two V100s.

Training

Pretraining: 16.7B tokens, fp16, AdamW (0.9, 0.95), weight decay 0.1, peak LR 8e-4, 262k tokens per step, warmup-stable-decay.

Anneal: 1.5B tokens while the learning rate decays to zero: mostly distilled Q&A and short facts, with explanations and short reasoning, Wikipedia, Cosmopedia (including WikiHow), FineMath, FinePDFs and short generated everyday scenes and stories.

Expert patch: factual Q&A. In each layer, the 4 experts that factual question-answer text uses most (relative to ordinary text) were trained for 131M tokens on plain-text Q&A, short facts and explanations, mixed with pretraining text. Everything else stayed frozen - attention, norms, embeddings, router and the other 20 experts per layer - and was checked bit-for-bit after every step. It is one model fine-tuned in place, not a merge. The patch raised ARC-Easy and ARC-Challenge without moving the other benchmarks (see "All benchmarks").

Limitations

• A small base model: fluent, but it states wrong facts with confidence, and it does not follow instructions.
• Not for code: the tokenizer collapses whitespace, so newlines and indentation are lost.
• English only.

Reproduce

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base,dtype=float32 \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16

TinyBrainBot v4.1 (mixture of experts)

• TinyBrainBot-360M-A75M-v4.1-MoE-Base (this model)
• TinyBrainBot-61M-A10M-v4.1-MoE-Base, the first MoE pilot