Nanoserve 27M

A 26,747,392-parameter decoder-only transformer trained from scratch on TinyStories V2. The architecture, the tokenizer and the training loop were all written by hand -- no AutoModel, no tokenizers library.

It writes short children's stories. It is not a general-purpose assistant and cannot answer questions, follow instructions, or reason. That is the specification, not a shortfall.

Results

Architecture

Parameter names mirror HuggingFace's Llama, so the weights load into a LlamaForCausalLM of matching shape.

Usage

from model.tokenizer import BPETokenizer
from model.generate import load_model, generate, SamplingParams

tok = BPETokenizer.load("tokenizer.json")
model, _ = load_model("model.safetensors", device="cuda")

print(generate(
    model, tok,
    "Once upon a time, there was a little girl named Lily who",
    SamplingParams(max_new_tokens=200, temperature=0.8),
))

Training data

TinyStories V2 (GPT-4 generated). 536,174,127 tokens under a purpose-built 8,192-token byte-level BPE vocabulary, which compresses this corpus to 3.965 bytes/token with 98.1% of words encoding to a single token.

Token count was chosen to sit at the Chinchilla-optimal ratio for the parameter count (~20 tokens per parameter).

Limitations

• Vocabulary and world model are limited to the simple English of TinyStories.
• 1,024-token context.
• No instruction tuning, no alignment, no safety filtering of any kind.
• Will confidently produce factually wrong statements; it models story-shaped text, not truth.

Source

Trained as Phase 1 of Nanoserve, a project that trains a small language model and then builds the distributed system that serves it.