This repo continues NeuroBLAST v46-150M from its pre-decay checkpoint (step 90,000, 6.5B tokens seen, learning rate still at its peak) up to 50B tokens, on a broader data mixture, with a final 32k-context stage. It trains on a single Colab A100. The chart and tables below update themselves at every eval point.
Work in progress. The
checkpoint-*folders are intermediate training states (model, optimizer, scheduler), not a finished model. Expect a base model that continues text, not a chat assistant.
Status (2026-11-15 03:30 UTC): step 465,119 of 465,119 (100.0%) · ~37.75B tokens seen · train loss 2.229
| step | tokens seen | eval loss | eval bits/byte | eval perplexity |
|---|---|---|---|---|
| 2,500 | 202M | 3.807 | - | 45.0 |
| 45,000 | 3.64B | 3.223 | - | 25.1 |
| 87,500 | 6.38B | 3.214 | 1.004 | 24.9 |
| 130,000 | 8.48B | 3.281 | 1.021 | 26.6 |
| 172,500 | 10.59B | 3.264 | 1.017 | 26.2 |
| 215,000 | 12.68B | 3.249 | 1.013 | 25.8 |
| 257,500 | 14.82B | 3.245 | 1.010 | 25.7 |
| 305,000 | 17.05B | 3.208 | 1.005 | 24.7 |
| 328,000 | 18.85B | 3.125 | 0.978 | 22.8 |
| 345,000 | 20.55B | 3.103 | 0.971 | 22.3 |
| 361,000 | 22.14B | 3.109 | 0.972 | 22.4 |
| 378,000 | 23.84B | 3.115 | 0.974 | 22.5 |
| 393,000 | 25.61B | 3.106 | 0.971 | 22.3 |
| 410,000 | 27.30B | 3.101 | 0.971 | 22.2 |
| 420,500 | 28.98B | 2.999 | 0.934 | 20.1 |
| 429,000 | 30.65B | 3.008 | 0.935 | 20.2 |
| 437,500 | 32.32B | 2.984 | 0.928 | 19.8 |
| 446,000 | 33.99B | 2.956 | 0.919 | 19.2 |
| 454,500 | 35.66B | 2.908 | 0.904 | 18.3 |
| 463,000 | 37.34B | 2.851 | 0.886 | 17.3 |
| 463,500 | 37.43B | 2.847 | 0.885 | 17.2 |
| 464,000 | 37.53B | 2.844 | 0.884 | 17.2 |
| 464,500 | 37.63B | 2.841 | 0.883 | 17.1 |
| 465,000 | 37.73B | 2.840 | 0.883 | 17.1 |
| 465,119 | 37.75B | 2.840 | 0.883 | 17.1 |
Data mixture, phase decay (non-pad tokens consumed since this launch, as of step 465,120)
| source | target share | consumed share | tokens |
|---|---|---|---|
| fineweb_edu | 31% | 31.0% | 390M |
| stack_edu | 27% | 27.0% | 340M |
| finepdfs_edu | 14% | 14.0% | 176M |
| dclm | 13% | 13.0% | 164M |
| finemath_4plus | 9% | 9.0% | 113M |
| finemath_3only | 3% | 3.0% | 38M |
| science_pdfs | 1% | 1.0% | 13M |
| gutenberg | 1% | 1.0% | 13M |
| pre1929_books | 1% | 1.0% | 13M |
Planned phases: stable from step 0 → decay from step 426,716 → long from step 465,120.
328 eval points in total; 20 evenly spaced ones and the newest five are shown (all of them: assets/training_log.json).
Updated automatically at every eval point. Tokens seen are summed from the trainer's num_tokens log (non-pad tokens), so they are approximate across resumes.
How to read the chart. The left part (up to step 90,000) is the first run: FineWeb-Edu only, 512-token context until step 70,424, then 2,048. From step 90,000 on, this run takes over from the same weights and optimizer state. The eval loss is measured on the same held-out FineWeb-Edu rows (2,048 tokens) throughout, so the curve is directly comparable across both runs. A small rise right after step 90,000 would not mean the model got worse: about two thirds of the new mixture is not FineWeb-Edu, so the model spends capacity on code, math, PDFs and books.
The first run used a warmup-stable-decay (WSD) learning-rate schedule. Its last checkpoint before the decay still has the peak learning rate and a "warm" optimizer, so training can simply go on from there on more tokens and then run its own decay, without repeating the first run's decay. The first run's own decayed final model ended at eval loss 2.875; this run's decay starts at step 811,820.
| Start | checkpoint-90000 of the first run (tag pre-decay-step-90000): weights, optimizer, scheduler |
| Token budget | 50B tokens in total (6.5B from the first run + 43.5B here), ~330 tokens per parameter |
| Schedule | peak LR 2e-3 continues to step 811,820, then a linear decay to 0 at step 992,274 (the last 20% of this run's steps) |
| Stage 1 | context 2,048, 48 rows per step, steps 90,000–946,419 (~41B tokens) |
| Stage 2 | context 32,768, 3 rows per step, steps 946,419–992,274 (~2.5B tokens of long documents), inside the decay |
| Optimizer | Muon on 2-D weight matrices, AdamW on everything else, no weight decay (unchanged) |
| Hardware | 1× NVIDIA A100 80 GB (Colab) |
The mixture follows an open-data recipe for English-first, text-only pretraining. It is built from FineWeb-Edu and DCLM, plus educational and scientific PDFs, code, math and public-domain books. Shares are of consumed tokens, not documents. The trainer sends each row to the source furthest below its share and logs the consumed shares, which appear in the table above.
| source | Hub dataset | stable phase | decay phase | 32k phase |
|---|---|---|---|---|
| Educational web | HuggingFaceFW/fineweb-edu (all dumps) |
36% | 31% | – |
| General curated web | mlfoundations/dclm-baseline-1.0 |
28% | 13% | – |
| Educational PDFs | HuggingFaceFW/finepdfs-edu (eng_Latn) |
9% | 14% | 35% |
| Scientific PDFs | allenai/dolma3_pool (olmocr_science_pdfs-*) |
6% | 1% | 35% |
| Educational code | common-pile/stackv2_edu_filtered (Stack-Edu score ≥ 3) |
12% | 27% | – |
| Math (high quality) | HuggingFaceTB/finemath finemath-4plus |
4% | 9% | – |
| Math (broader) | HuggingFaceTB/finemath finemath-3plus, score 3 only |
3% | 3% | – |
| Books | common-pile/project_gutenberg_filtered |
1% | 1% | 15% |
| Books | common-pile/pre_1929_books_filtered |
1% | 1% | 15% |
data_mix.json in the code payload), so the stream can be reproduced.Checkpoints load like those of the first run. Take the model code from its src/ folder
(trust_remote_code=True) and the weights from a checkpoint-* folder here. A ready-to-load src/
snapshot will be added here as training progresses.
Only the newest checkpoints stay on the main branch. The trainer never deletes these two, and the watch also tags them:
| tag | what it is |
|---|---|
pre-decay-step-<N> |
last checkpoint before this run's learning-rate decay (step 811,820): continue from here with more compute |
stage1-end-ctx2048-step-<N> |
end of the 2,048-token stage, just before the 32k stage |
This is a small research model trained on web text, PDFs, code, math and old books. It knows little, makes facts up, can reproduce biases in its training data, and has had no instruction tuning or safety tuning. Use it for research and learning, not for decisions.
checkpoint-426000/optimizer.pt 810 MBcheckpoint-465000/optimizer.pt 810 MBcheckpoint-465119/optimizer.pt 810 MBcheckpoint-426000/model.safetensors 607 MBcheckpoint-465000/model.safetensors 607 MBcheckpoint-465119/model.safetensors 607 MBcheckpoint-426000/rng_state.pth 14.6 kBcheckpoint-465000/rng_state.pth 14.6 kBcheckpoint-465119/rng_state.pth 14.6 kBcheckpoint-426000/training_args.bin 6.9 kBcheckpoint-465000/training_args.bin 6.9 kBcheckpoint-465119/training_args.bin 6.9 kBcheckpoint-426000/scheduler.pt 1.5 kBcheckpoint-465000/scheduler.pt 1.5 kBcheckpoint-465119/scheduler.pt 1.5 kB