tiny-edu-114m

A 114M-parameter decoder-only language model trained from scratch (the author's second language model, after the 42M one and before Zenyx-100M-Base). It uses the GPT-2 tokenizer, which wastes much of the capacity on a 50k vocabulary that is poorly suited to its size (the embedding table alone is 38.6M of the 114M parameters); the project moved to a custom 49k tokenizer afterwards. The weights are published for completeness and for comparison; this is a base model, not instruction-tuned, and it is not recommended for use.

Previously named tiny-edu-50m; the "50m" did not match the real size (114.1M parameters). The old name redirects here.

At a glance

The architecture was reconstructed from the checkpoint tensors (the original training code is not in this repository). The reconstruction reproduces the validation loss stored in the checkpoint: 3.82 on FineWeb-Edu text against the recorded 3.817, so the implementation is right. The exact training data mixture is not documented here; the model name and the loss on FineWeb-Edu point to an educational-web corpus.

Results

Base model, 0-shot, all test items, bfloat16 weights run in fp16. Same harness and items as the other Zenyx models (Fact1000 is a 1,000-item four-way factual test written by the author, chance 25%). Standard error is about 1.5 points on 1,000 items.

Bits per byte does not depend on the tokenizer, so the Pile rows are a fair comparison. Zenyx-100M-Base scores much higher with a tenth of the training tokens. The likely causes are the tokenizer and the data mixture (not tested separately).

Usage

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoTokenizer
from modeling_tiny_edu import TinyEdu, generate      # download modeling_tiny_edu.py from this repository

model = TinyEdu(); model.load_state_dict(load_file(hf_hub_download("Arko007/tiny-edu-114m", "model.safetensors")), strict=False)   # strict=False: the output head is tied
model = model.float().eval()
tok = AutoTokenizer.from_pretrained("gpt2")
print(generate(model, tok, "The capital of France is", max_new=30))

Sampled continuations (temperature 0.8, top-k 50, one sample each; generated with the code above):

The capital of France is Germany, and the largest city in southern France is Germany. The French state is the biggest city in France, with the capital of France a third. With its capital, the city is home to

Photosynthesis is an important role in the production of biomass. In particular, it has been suggested that the uptake of biomass from the soil is the key to improving plant productivity. At the same time, biomass contains essential

In 1969, the first humans to be fully independent were humans. “If we were to be independent, we would have to be independent” (Battles, p. 39). In 1974, the Human Rights

To make a good cup of tea, you should have a cup of coffee in the middle of the cup, and a cup of coffee that you can use for a week. A cup of coffee can be used to make a cup of tea or

Limitations

• Base model: it continues text, makes up facts, repeats itself in long outputs, and does not follow instructions.
• English only, GPT-2 tokenizer (poor on code, digits and non-English text), not evaluated for bias or harmful content.
• Numbers on this card come from one checkpoint and the author's own harness; Fact1000 is written by the author.
• The original training code and data mixture are not published, and 633 intermediate checkpoints (iterations 473,000 to 790,000) were removed from this repository to keep it small; only the final weights remain.