This is the control for adyaanahmed/distil-50m, a 51.5M-parameter model distilled from Qwen2.5-0.5B. It has the same architecture, data, data order, seed, optimiser, number of steps and tokens, but it is trained with plain next-token cross-entropy and no teacher, both in pretraining and in instruction tuning. The point is to measure what the teacher actually buys.
It is a research artefact, not a good assistant: it is fluent but invents facts and cannot do arithmetic.
| distil-50m (distilled) | distil-50m-no-teacher (this repo) | |
|---|---|---|
| Architecture | 51,532,544 params, 18 layers, d_model 512, GQA 8:2, QK-norm, SwiGLU, RoPE, tied factorised embedding, 512 context | identical |
| Pretraining data | ~1.02B tokens: Cosmopedia v2 + FineWeb-Edu (score >= 3) | identical, same order |
| Steps / tokens | 15,000 x 65,536 = 983M tokens | identical |
| Optimiser / schedule | Muon + AdamW, lr 1.5e-3, warmup-stable-decay, seed 0 | identical |
| Pretraining loss | 0.7 x KL(teacher) + 0.3 x CE, teacher Qwen2.5-0.5B | CE only, no teacher |
| SFT data | 150k smol-smoltalk first exchanges (about 130k kept) | identical |
| SFT loss | 0.5 x KL(Qwen2.5-0.5B-Instruct) + 0.5 x CE | CE only |
| Pretraining compute | 60.5 T4 GPU-hours (9.9k tokens/s) | 20.0 T4 GPU-hours (27.8k tokens/s) |
| SFT compute | 5.7 GPU-hours | 2.3 GPU-hours |
| Total compute | 66.2 GPU-hours | 22.3 GPU-hours (3.0x less) |
Held-out bits per byte (lower is better):
| Set | Distilled | No teacher | Teacher (Qwen2.5-0.5B) |
|---|---|---|---|
| Cosmopedia (in-domain) | 0.666 | 0.655 | 0.572 |
| FineWeb-Edu | 1.078 | 1.117 | 0.835 |
| WikiText (out of domain) | 1.270 | 1.325 | 0.905 |
Zero-shot benchmarks, base (pretrained) models, 1,000 examples each (about +/-1.5 points of noise):
| Benchmark | Distilled | No teacher |
|---|---|---|
| ARC-Easy (acc_norm) | 45.1 | 42.8 |
| SciQ (acc) | 67.4 | 62.8 |
| HellaSwag (acc_norm) | 30.6 | 28.4 |
| LAMBADA (acc) | 23.0 | 20.9 |
| LAMBADA perplexity (lower is better) | 185 | 303 |
Chat (instruction-tuned) models:
| Benchmark | Distilled | No teacher |
|---|---|---|
| ARC-Easy (acc_norm) | 41.9 | 38.0 |
| SciQ (acc) | 62.0 | 56.2 |
| HellaSwag (acc_norm) | 29.6 | 27.9 |
| LAMBADA (acc) | 16.3 | 12.5 |
Same layout as the distilled repo: student_chat.safetensors (instruction-tuned) and student_pretrained.safetensors (base), both fp16, 99 MB each; vocab.json; hybrid50m/ (model code); chat_local.py; report.json; training dashboards.
pip install torch safetensors tokenizers
huggingface-cli download adyaanahmed/distil-50m-no-teacher --local-dir distil-50m-no-teacher && cd distil-50m-no-teacher
python chat_local.py # instruction-tuned model
python chat_local.py --model pretrained # raw text continuation
The chat prompt format is ### Question:\n{query}\n\n### Answer:\n, single turn, 512-token context. The tokenizer is Qwen2.5's; vocab.json maps its ids to the student's 32,768-token vocabulary.
Limitations are the same as for the distilled model, only slightly worse: it drifts and repeats in long answers (the instruction-tuned LAMBADA perplexity is 6,001 versus 882 for the distilled model), invents facts, cannot do arithmetic, and is not safety-tuned. English only.
Apache-2.0. Data: Cosmopedia v2 and smol-smoltalk (Apache-2.0), FineWeb-Edu (ODC-By 1.0). No teacher model was used to train these weights.
student_chat.safetensors 103 MBstudent_pretrained.safetensors 103 MB