AtoMinima
A 420-parameter character-level GPT. Trained on 370,000,000 characters of FineWeb-edu (sample-10BT).
Benchmarks
Zero-shot, scored as mean log-likelihood per byte over each answer choice, argmax of the choices. Full splits.
| Benchmark | AtoMinima | Chance |
|---|---|---|
| HellaSwag | 25.53% | 25.00% |
| ARC-Easy | 26.98% | 25.00% |
| ARC-Challenge | 26.02% | 25.00% |
| PIQA | 49.13% | 50.00% |
| ArithMark-3.0 | 30.70% | 25.00% |
| Open SLM index | +1.62 | 0.00 |
Held-out validation loss: 3.8371 bits/byte (1M characters of FineWeb-edu held out from training).
Model details
| Component | Details |
|---|---|
| Architecture | decoder-only transformer, pre-norm, causal |
| Layers | 1 |
| Width | d_model 4, 1 head, head_dim 4 |
| Feed-forward | 4 -> 16 -> 4, GELU |
| Normalization | LayerNorm |
| Embeddings | tied input/output, 42 ids |
| Positions | fixed sinusoidal |
| Dropout | none |
| Parameters | 420 trainable |
| Context | 256 characters |
| Tokenizer | character-level: a-z 0-9 space . , ' \n + unknown |
| Training tokens | 370,000,000 |
| Training data | FineWeb-edu sample-10BT, 10M-character slice |
| Checkpoint size | 11 KB |
Usage
pip install torch huggingface_hub
python example.py
import atominima
model = atominima.from_pretrained("arthurblg1802/AtoMinima") # needs atominima.py in the path
print(model.generate("the water cycle begins when", max_new_tokens=80))
atominima.py in this repo holds the whole model and tokenizer: 90 lines, torch only, no other dependency.
Limitations
This is a character-level model with 420 parameters: it has no usable language ability, and every benchmark score above sits within noise of its baseline except ArithMark-3.0, which is below the 50.80% floor reachable by always answering the second-smallest number. Generated text is not English. It exists as a lower bound, not as a tool.
Licence
Apache-2.0.