TinyBrainBot 303M V2 — Base
A ~303M parameter language model pretrained from scratch on a single 2×GPU workstation. This is the base (completion) model — no instruction tuning. For the chat variant, see nkthebass/tinybrainbot-303mV2-instruct.
Use this if you want a small English foundation model to continue text or fine-tune yourself.
Architecture
Llama-family (RoPE, RMSNorm, SwiGLU, GQA), tied embeddings.
| Parameters | ~303M |
| Hidden size | 1024 |
| Layers | 24 |
| Attention heads | 16 (4 KV heads, GQA) |
| Head dim | 64 |
| FFN size | 2816 (SwiGLU) |
| Vocab | 32,000 (SentencePiece BPE) |
| Context length | 1024 |
| RoPE theta | 10000 |
Usage
transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-303mV2-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-303mV2-base")
ids = tok("Once upon a time, in a small village", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
llama.cpp / Jan / LM Studio
GGUF files (F16, Q8_0) are included in this repo.
Training data
Pretrained (fp16, WSD schedule, DDP on 2× Tesla P100) on a mix of:
- FineWeb-Edu (10BT sample) — filtered educational web text
- Wikipedia (English)
- TinyStories v2 — simple synthetic stories (grammar/coherence)
- OpenWebText2
- Orca-Math — grade-school math word problems
- Synthetic distillation sets (generated by a larger teacher): short-answer facts + longer general Q&A
Note: because the pretraining mix included some chat-formatted synthetic Q&A, the base will occasionally continue a completion into <|user|>/<|assistant|>-style turns. That's expected for this base; the instruct model is the one tuned to actually chat.
Benchmarks
Multiple-choice log-likelihood scoring puts the family at roughly GPT-2-124M class on QA benchmarks (ARC-Easy ~46, ARC-Challenge ~27, beating GPT-2-124M on both), and at the random floor on HellaSwag/MMLU/WinoGrande — the size ceiling of a 303M. Full numbers are on the instruct card (multiple-choice scores barely move between base and instruct).
Limitations
- 303M parameters — it hallucinates and knows only common facts.
- English only, 1024-token context.
- No instruction tuning, no safety tuning. This is a raw base model.
License
Apache-2.0. Free to use and build on — attribution appreciated. Trained on public/open datasets (FineWeb-Edu, Wikipedia, TinyStories, OpenWebText2, Orca-Math, plus synthetic distillation data); please respect the licenses of those upstream sources.