memcap: 1.86M-param GPT-2, saturation (capacity plateau) regime

Part of a tiny-scale reproduction of How much do language models memorize? (Morris et al., ICML 2026 honorable mention): GPT-style models trained from scratch on uniformly random token sequences, where generalization is impossible and every bit of loss reduction is memorization.

This checkpoint sits at the model's saturation point: memorization has plateaued at 3.50 bits per parameter, the headline capacity measurement of this reproduction and the closest match to the paper's ~3.6 bits/param (fp32: ~3.83). Dataset entropy (26.3 Mbits, ~3.9x the saturation knee) exceeds capacity, so the model stores only what it can.

Model

• Architecture: GPT-2 (via transformers), 1.86M parameters (d_model 128, 8 layers, 4 heads), vocab 2048, context 64, trained from scratch (init seed 0)
• Precision: fp32 (TF32 disabled) | Optimizer: AdamW lr 1e-3, constant LR | Batch: 1024 sequences
• Trained for 37,750 steps (converged: extrapolated memorization growth < 0.3% over 2,000 steps)

Data

Uniformly random tokens, vocab 2048, 64 tokens/sequence, N = 38,000 sequences, data seed 200. Dataset entropy = 63 predictable tokens x log2(2048) = 693 bits/sequence = 26,334,000 bits total. Dataset: pngwn/memcap-random-data

Memorization measurement

memorized bits = dataset entropy - model NLL on the data (in bits)

Full learning curves and all 18 cells: pngwn/memcap-results. Training script: train_memcap.py in that repo. This same checkpoint also lives on branch m_n38000_s0_fp32 of pngwn/memcap-runs.

Usage

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("pngwn/memcap-m-near-capacity")

Note: no tokenizer ships with this checkpoint — the training data is raw token ids in [0, 2048).