lecore-303m-rerun — matched 303M pair: softmax vs gated-HRR

Result: gated-HRR attention is 41.5% worse than softmax at 303M parameters / 1B tokens. The gap did not narrow with scale, it widened (at 12M params on enwik8 it was ~14%).

Two arms, identical in everything but the attention operator: same seed, same 7,629 steps, same cosine schedule, 131,072 tokens/step, 5,000,002,551-token FineWeb-Edu corpus (GPT-2 BPE), RoPE in both arms, block 1024. This is the run that answers the "does the from-scratch gap close at scale?" question the earlier postmortem left open.

HRR / softmax = 1.415. Speed, measured on an H100 with the repo's Triton scan wired in: softmax 151,269 tok/s, gated-HRR 50,797 tok/s — 3.0x slower, not 11x (the Triton kernel is a 10.2x speedup over the naive path and cut memory 74.9 → 41.8 GB).

Files

runs303/ and runs6b/ are the two run sets, each with an hrr/ and a softmax/ arm and both final.pt and latest.pt (identical when the run finished cleanly). Checkpoints are ~1.4 GB each.

What this does and does not say

• It does not say HRR is useless: it is a fixed-size recurrent state with O(T) cost. The loss gap is the price; whether the price is worth paying depends on the workload.
• It does not measure long-context behaviour — both arms trained at block 1024.
• n=1 per arm. Single run, no seed sweep.

Related

• Method, tiny-twin benchmarks, constant-memory footprint numbers: staccs/lecore-hrr-gpt2
• Per-model gated-HRR gate sets: Qwen3-32B, Llama-3.3-70B, QwQ-32B, Qwen2.5-72B, Apertus-70B
• Training harness and corpus prep: https://github.com/staccDOTsol/supercontext

License

MIT — checkpoints and code produced here, no third-party base weights.