TinyBrainBot-360M-A75M-v4.1-MoE-Instruct

Superseded by TinyBrainBot-360M-A75M-v5-Instruct, which adds math and reasoning and scores higher on IFEval (25.1 vs 18.5).

Try it in your browser: the Q8_0 GGUF runs locally on your device with WebGPU, and nothing you type is sent anywhere.

The chat version of TinyBrainBot-360M-A75M-v4.1-MoE-Base: a mixture-of-experts model trained from scratch on two Tesla V100s, with 24 experts per layer and 4 used per token (359M parameters stored, 75.7M active per token, 383M in total with the embedding). It holds a multi-turn conversation, remembers what you said earlier in the chat, follows format instructions much better than our earlier chat models, works arithmetic step by step, and keeps a correct answer when you push back.

Full test sets, zero-shot, acc_norm, EleutherAI lm-eval-harness, plus the official ArithMark-3 script; the Intelligence Index is the Open SLM Leaderboard's. TinyBrainBot rows were run by us; the others are the leaderboard's published numbers.

Instruction following (IFEval)

lm-eval ifeval (541 prompts with checkable instructions such as word limits, required words or formats), chat template applied, greedy decoding; both models run by us the same way. SmolLM2-360M-Instruct was trained on far more data (4T pretraining tokens); this model is still well behind it here.

Quick start

A standard Mixtral model with a chat template, no custom code:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nkthebass/tinybrainbot-360m-a75m-v4.1-moe-instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

msgs = [{"role": "user", "content": "What is 23 plus 19?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=200)        # defaults from generation_config.json
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Ollama): tinybrainbot-360m-a75m-v4.1-moe-instruct-Q8_0.gguf (409 MB) or -F16.gguf (768 MB). The chat template is embedded and both tokenize exactly like the transformers version.

llama-cli -m tinybrainbot-360m-a75m-v4.1-moe-instruct-Q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf

Recommended sampling: temperature 0.7, top_p 0.9, repetition_penalty 1.15. The two --logit-bias flags block the byte tokens that start every emoji (the tokenizer has no emoji pieces, so the model would spell them out byte by byte); generation_config.json does the same for transformers via bad_words_ids.

Conversations

Unedited, sampled at the recommended settings; each user turn was sent after the model's real reply. Five of the twelve test conversations are shown here; all twelve, including the ones it gets wrong, are in conversation_samples.md.

Remembering me

User: Hey, I'm Sam. I'm 15 and I play basketball.

Bot: Nice to meet you! How can I help?

User: What's a good drill to get better at shooting?

Bot: A basic beginner drill is best: shoot a ball at eye level, keep your elbow under the ball and with body pointing through the ball's length, and aim for the top of the arc.

User: How old did I say I was?

Bot: You said you're 15.

User: And what sport do I play?

Bot: You said you play basketball.

Math, step by step

User: What is 23 plus 19?

Bot: Adding. ones: 3 + 9 = 12, write 2 carry 1. tens: 2 + 1 + 1 (carry) = 4, write 4. The answer is 42.

User: Now multiply that by 2.

Bot: Multiplying. Break 2 into digits. 42 x 2 = 84. The answer is 84.

User: Is that number even or odd?

Bot: 84 is even, because it ends in 4.

Pushback

User: What's the capital of Australia?

Bot: Canberra is the capital of Australia.

User: Are you sure? I thought it was Sydney.

Bot: That's a common mix-up. The capital of Australia is Canberra. Sydney is the largest city, but Canberra is the capital.

Recommendations

User: Can you recommend a book for a teenager?

Bot: Try 'The Martian' by Andy Weir. It's a fun survival story about an astronaut stranded on Mars, perfect for science and problem-solving.

User: Something shorter?

Bot: How about 'The Little Prince' by Antoine de Saint-Exupery? It's a short, simple tale about friendship and seeing the world differently.

User: What is it about?

Bot: It follows a young prince from a tiny asteroid who meets a stranded pilot in the desert. Through their talks, he learns about love, loss, and what truly matters to him.

Stress

User: I'm really stressed about my exams next week.

Bot: Exams can feel overwhelming. Is there one topic you're most worried about?

User: What if I study and still fail?

Bot: Focus on the topic you're good at and take breaks. You can also review notes from your classmate or teacher for inspiration.

User: Thanks, that actually helps.

Bot: You're welcome. Small steps count.

Arithmetic in chat (generated, 25 fresh problems each, greedy): 2-digit addition 100%, 2-digit subtraction 100%, 3-digit addition 100%, 3-digit subtraction 96%, 4-digit addition 100%. Multiplication is weak (4% on 2-digit x 2-digit).

All benchmarks

lm-eval-harness, 0-shot, up to 2,000 examples per task (HellaSwag and MMLU use their first 2,000; the table at the top uses full sets). Each task's first row is acc_norm where reported, acc otherwise; the "(raw acc)" row is acc.

How it was trained

Starting from the pretrained model (18.2B tokens, see the base card):

1. Expert patches. Factual Q&A and worked arithmetic were each trained into 4 experts per layer with every other weight frozen (checked bit-for-bit).
2. Chat SFT, following the recipe of SmolLM2-360M-Instruct: a full fine-tune for ~250M tokens at peak LR 2e-4 on smol-smoltalk (SmolTalk for sub-1B models), Tulu 3 persona instruction following, filtered short conversations from WildChat-4.8M, SmolTalk2 everyday conversations, multi-turn conversations we generated with DeepSeek V4 Flash (follow-ups, corrections, recommendations, everyday feelings), exact-quote recall and "double-check" chats, and a replay of worked arithmetic and short facts.
3. DPO (beta 0.5, LR 1e-6, as SmolLM2): ~10k preference pairs - the model's own replies (6 per prompt, single-turn, multi-turn and instruction-following prompts) ranked twice by DeepSeek V4 Flash with length kept equal, instruction-following pairs decided by an exact constraint checker, DeepSeek-written replies against the model's weakest, and a short-reply slice of UltraFeedback.

Boundary. The only refusal training is narrow: requests to kill or seriously hurt people get a short refusal, and self-harm messages get a supportive reply. On held-out prompts it refuses 85% of violence/self-harm requests and 2% of harmless prompts. There is no other safety training.

Limitations

• A 75M-active model: it makes up facts, and long conversations drift.
• Creative edits ("make it funnier") and multiplication are weak.
• Not for code: the tokenizer collapses whitespace, so newlines and indentation are lost.
• English only.

Data attribution

WildChat-4.8M and the Tulu 3 persona instruction-following set (both AllenAI) are released under ODC-BY; SmolTalk / smol-smoltalk / SmolTalk2 under Apache 2.0; UltraFeedback under MIT.

TinyBrainBot v4.1 (mixture of experts)

• TinyBrainBot-360M-A75M-v4.1-MoE-Instruct (this model)
• TinyBrainBot-360M-A75M-v4.1-MoE-Base
• TinyBrainBot-61M-A10M-v4.1-MoE-Base