tinyllm-113m-base
A 112,678,656-parameter English base language model built on the stock LlamaForCausalLM
architecture. It was pretrained from scratch on 15B tokens of FineWeb-Edu on a single
NVIDIA L4 (about 201 hours). This is a text-completion model, not a chat model. The
chat version is sciencemj/tinyllm-113m-chat,
which has the full model card.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "sciencemj/tinyllm-113m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()
enc = tok("The water cycle begins when", return_tensors="pt")
out = model.generate(**enc, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
Model and training
- 14 layers, hidden 768, SwiGLU FFN 2,048, 12 query / 4 KV heads (GQA), RoPE, RMSNorm, tied embeddings, 32,000-token BPE vocab, 1,024-token context.
- FineWeb-Edu
sample-100BT: 15.0B tokens (915,528 steps × 16 × 1,024), about 0.75 of a 20.0B-token sample. - AdamW (β 0.9/0.95), LR 6e-4 → 6e-5 cosine with 2,000 warmup steps, weight decay 0.1, grad clip 1.0, bf16 mixed precision.
- This is the checkpoint with the best validation loss (step 879,999): 3.1298 nats/token, 1.0021 bits/char on 22.2M held-out FineWeb-Edu tokens.
Limitations
It continues text in the style of educational web pages. Given a story opening, it often drifts into encyclopedic prose. It repeats itself at times, and its facts are unreliable. English only, with no safety tuning.
Data and license
HuggingFaceFW/fineweb-edu,
ODC-By 1.0, plus the Common Crawl terms of use. The weights are released under Apache-2.0.