TinyQwen3-1.8M-Base

TinyQwen3-1.8M-Base is a small causal language model using the Qwen3 dense architecture.

The tokenizer and model weights were pretrained from scratch on the 50k configuration of TinyStories-Tiered, a cleaned, deterministic and tiered edition of TinyStories. No pretrained Qwen weights were used.

Model specification

The repository contains an FP16 checkpoint at the root, with BF16 and FP32 checkpoints in the bf16/ and fp32/ subdirectories. The tokenizer is shared by all checkpoints and is loaded from the root.

Training

The training corpus is the train split of the 50k tier: 50,000 stories, about 14.66M tokens including one EOS per story as measured by the dataset's reference tokenizer. All tokens are packed into 256-token windows with one-token overlap, and the 50 epochs represent approximately 733M predicted-token exposures over the same training subset.

Each run trains a fresh 1,024-token byte-level BPE on the first 2,048 training stories. Because tiers are nested prefixes, this tokenizer sample is identical for every tier from 2k upward.

To reproduce the recorded settings, run from this directory:

python train.py --tier 50k --epochs 50 --seed 1234

Checkpoints, the tokenizer, and metrics.json are written to the current working directory by default. Use --output-dir to choose another folder.

Evaluation

Training evaluates the tier's validation split after every epoch, and the best checkpoint is selected on it. The held-out test split is evaluated once on the best checkpoint and recorded as best_test.

The following results come from reloading and evaluating each serialized checkpoint on the packed validation split:

Cross-entropy is token weighted; perplexity is its exponential. These metrics use the model's 1,024-token tokenizer and 256-token context. The first token of the validation corpus supplies context rather than a prediction target.

Validation loss fell steadily and reached its minimum at epoch 49, the retained checkpoint:

Full epoch history, checkpoint evaluations and the held-out test result are available in metrics.json.

Attributions

Architecture

The model uses the Qwen3 architecture but does not contain pretrained Qwen weights.

Qwen3 is described in Qwen3 Technical Report, Yang et al. (2025). The model implementation is provided by Hugging Face Transformers.

Dataset

TinyStories-Tiered is a cleaned, deduplicated and tiered edition of TinyStories, derived from the work of Ronen Eldan and Yuanzhi Li, authors of TinyStories: How Small Can Language Models Be and Still Speak Coherent English?.

The source dataset is licensed under CDLA-Sharing-1.0. The dataset is loaded from Hugging Face; model exports contain no dataset text.