A 112,678,656-parameter English base language model built on the stock LlamaForCausalLM
architecture. It was pretrained from scratch on 15B tokens of FineWeb-Edu on a single
NVIDIA L4 (about 201 hours). This is a text-completion model, not a chat model. The
chat version is sciencemj/tinyllm-113m-chat,
which has the full model card.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "sciencemj/tinyllm-113m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()
enc = tok("The water cycle begins when", return_tensors="pt")
out = model.generate(**enc, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
sample-100BT: 15.0B tokens (915,528 steps × 16 × 1,024), about 0.75 of a
20.0B-token sample.It continues text in the style of educational web pages. Given a story opening, it often drifts into encyclopedic prose. It repeats itself at times, and its facts are unreliable. English only, with no safety tuning.
HuggingFaceFW/fineweb-edu,
ODC-By 1.0, plus the Common Crawl terms of use. The weights are released under Apache-2.0.