ChonkyKedi-nano-base

ChonkyKedi-nano-base is a small dense Llama-style language model trained from scratch. It has 25.2M parameters (12.6M excluding the embedding table) and was pretrained on a 1.00B-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, SmolTalk, Wikipedia (English, 2023-11-01), github-code-clean).

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The ChonkyKedi family

ChonkyKedi models share one architecture, tokenizer and data pipeline, and differ only in scale. Each is trained on about 80 tokens per non-embedding parameter. The table is generated from the current presets.

Architecture

A dense, Llama-style decoder-only transformer (llama_gqa).

Training

79 documents were dropped by decontamination in total.

Every training document was checked against every item of the five evaluation sets below (10-gram overlap, plus exact matches of short answers) and dropped on a hit. The validation set comes from shards the training stream never reads.

Evaluation

Validation loss over training: 10.48 (0.00B tok) → 4.16 (0.15B tok) → 4.11 (0.29B tok) → 4.07 (0.43B tok) → 4.05 (0.57B tok) → 4.03 (0.72B tok) → 4.03 (0.86B tok) → 3.68 (1.00B tok)

Zero-shot benchmarks (main.py --mode bench)

Each answer choice is scored by the log-probability of its text after the prompt (BLiMP: the whole sentence of each minimal pair). acc_norm divides by the answer's length in UTF-8 bytes; acc doesn't. Every item of each set is scored, not a sample; ± is one standard error.

Compared with other small models

ChonkyKedi columns are measured with this repo's evaluator on the checkpoints named; validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts, splits and length normalisation differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the smallest ChonkyKedi: a dense 25.2M-parameter model (12.6M excluding the embedding table) trained from scratch on a laptop's integrated GPU, reaching a validation loss of 3.676 after 1.00B tokens. It writes grammatical, on-topic English sentences and simple children's stories, but it has almost no world knowledge and it repeats itself heavily. It's a research and educational artifact, and a baseline for the larger ChonkyKedi models (mini, at 2.9× the parameters and 4.1× the data, is better on every benchmark below), not a source of information.

What it does

• ✅ Writes grammatical sentences that stay on the prompt's topic. Grammar is its strongest skill: 67.7% on BLiMP across all 67,000 minimal pairs.
• ✅ Tells simple children's stories in the style of its synthetic-textbook data: "Once upon a time, a little cat" → "named Lily was playing with her friends… One day, while playing outside, Lily noticed something strange." This is the most consistently good output it has.
• ✅ Gets the rough shape of a few very common topics: "The heart pumps" → "blood, providing oxygen and food for the body" (one of four tries); "Photosynthesis is" → "a process of converting the energy from the sun into energy"; "Albert Einstein was" → "a brilliant scientist who was known for his groundbreaking discoveries in the field of physics".
• ✅ Knows the chat format. About 5% of its pretraining data was SmolTalk conversations, so given <|user|>…<|assistant|> it replies in the assistant's voice, often ending with <|eos|>, as a base model rather than an instruction-tuned one.
• ✅ Zero-shot benchmarks (acc_norm, every item scored): SciQ 62.7%, ARC-Easy 36.8%, PIQA 56.9%, HellaSwag 26.7% (close to its 25% chance level). That's above Pythia-70M on SciQ and ARC-Easy and above GPT-2 on SciQ, from 1.00B training tokens, but below GPT-2 on ARC-Easy and HellaSwag; prompts and normalisation differ between published numbers and ours.

What it can't do

• ❌ Facts. It got none of the factual prompts right in four tries each (greedy plus three seeds): "The capital of France is" → "the capital of France", "the state of Paris"; "Water boils at" → "the bottom of the pond"; "The largest planet in the solar system is" → "the Sun"; "The chemical symbol for gold is" → "the symbol for gold"; "The French Revolution began in" → "1776", "1849"; "Albert Einstein was" → "born in 1907… a student at the University of Michigan". Treat every name, date and number as made up.
• ❌ Arithmetic. "2 + 2 =" → 2, 90; "7 + 6 =" → 12, 109, 6, 90. A train going 60 miles in 2 hours does "100 miles per hour" or "90 miles per hour".
• ❌ Code. It writes Python-shaped lines but nothing that works: def fibonacci(n): → if n < 0: return fibonacci(n), or the definition repeated.
• ❌ Instructions. No instruction tuning. Asked for a haiku about the sea, it writes a prose sentence ("The sea is a lovely sea, and the sea is a vast ocean."); asked for the largest planet, it says the Sun, the Kuiper Belt or the Solar System.
• ❌ Causal reasoning and history. "The main causes of the First World War were" → "the loss of supplies, the loss of land, and the loss of the land" or a 1949 battle.
• ❌ Coherence beyond a sentence or two. It restates the prompt ("The capital of France is the capital of France"), repeats words within a sentence ("vibrant, and vibrant"), and drifts off topic within a paragraph.
• ❌ Long context. It was trained on 1,024-token sequences; nothing longer was tested.

Decoding

Greedy decoding loops almost immediately ("The symbol for gold is the symbol for gold. The symbol for gold is the symbol for gold…"; "The water is in the water, and the water is in the water…"). Use sampling with temperature ≈ 0.7 and top-k ≈ 40, the defaults of ChonkyKediModel.generate, and expect to try several seeds.

Usage

PyTorch (needs the chonkykedi package from the training code):

import torch
from chonkykedi.models.model import ChonkyKediModel
from chonkykedi.tokenizer.tokenizer import ChonkyTokenizer

model = ChonkyKediModel.from_pretrained(".")  # this folder: config.json + model.safetensors
tok = ChonkyTokenizer.load("tokenizer.json")
ids = torch.tensor([tok.encode("Photosynthesis is", add_bos=True)])
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Files

Acknowledgements

• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).

Card generated 2026-11-02 by prepare_release.py from llama_gqa_nano_1b.pt.