TinyBrainBot-100M-v5.1-Instruct

A 100M chat model that can finally use line breaks. Dense, 100.1M parameters, trained from scratch, and loads as a plain LlamaForCausalLM (no custom code). It chats, follows instructions, remembers what you said earlier in the conversation, and works through arithmetic word problems in a visible <think> scratchpad.

Earlier TinyBrainBot models on this tokenizer never saw a line break, so lists, poems and steps came out as one long line. This one was trained with them: numbered lists and poems come out on separate lines.

Scores

IFEval was run with lm-eval and the chat template, greedy, with the <think> block removed before scoring. The Index uses the full HellaSwag / ARC / PIQA sets plus ArithMark-3, 0-shot.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-100m-v5.1-instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
msgs = [{"role": "user", "content": "Write a short poem about autumn."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=200, temperature=0.4)   # other defaults from generation_config.json
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Jan, Ollama): F16 (201 MB), Q8_0 (107 MB) and Q4_K_M (66 MB). The chat template is embedded, chats tokenize exactly like transformers (line breaks included), and greedy output matches token for token.

llama-cli -m tinybrainbot-100m-v5.1-instruct-Q8_0.gguf -cnv --temp 0.4 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf

Recommended sampling: temperature 0.4, top-p 0.9, repetition penalty 1.15. The two --logit-bias flags block the byte tokens that start every emoji: the tokenizer has no emoji pieces, so the model would otherwise spell them out byte by byte.

<think> blocks are plain text, not special tokens. Strip them if you only want the answer:

import re; answer = re.sub(r"<think>.*?</think>", "", reply, flags=re.S).strip()

What it does well (and doesn't)

• Conversation: short, friendly answers; it remembers names and details from earlier turns.
• Layout: numbered lists and multi-line poems.
• Math: it thinks before answering on 9 of 10 short math questions, and gets about half of them right.
• Not yet: it doesn't always follow exact counts ("give me 3 tips" can give 5), some practical answers drift off topic, and at 100M parameters it makes up facts.

Unedited sample conversations, including mistakes, are in conversation_samples.md.

Training

1. Pretraining. TinyBrainBot-100M-v5.1-Base: 29.6B tokens.
2. SFT (0.39B tokens, 3,000 steps), one mix for every skill, the recipe from 360M-A75M-v5-Instruct, now with line breaks kept:

3. DPO. 10,383 preference pairs (UltraFeedback, instruction-constraint pairs checked by code, and judge-ranked pairs), beta 0.5, LR 1e-6, 2 epochs.

No benchmark training or test data was used, and no GSM8K-derived sets.

Limitations

• English only. It confabulates facts, so check anything important.
• Extra spaces are still collapsed, so it can't keep code indentation.
• 1,024-token context.