Apertus-70B-Instruct-2509-lecore-hrr — leCore gated-HRR attention gates (WORK IN PROGRESS)

Work in progress — not a working model yet. Passkey recall still fails; what is verified is the constant-memory claim (a fixed 3.3 MB attention state on the small proof). These are the trained gate tensors plus the code that produced them, published so the run is reproducible and the checkpoint is not lost.

What this is

A drop-in replacement for full attention in swiss-ai/Apertus-70B-Instruct-2509: the layer keeps a sliding-window softmax (fixed window W=256) plus a gated HRR (holographic reduced representation) recurrent state, so decode memory is O(1) in context length instead of a KV cache that grows with it. code/hrr_swa.py holds the module (HRRSWAAttention(old, cfg, window=256)), code/gated_scan_triton.py the Triton scan kernel, code/bigjob.py the training driver: it loads one copy of the model, toggles the original attention as the teacher, trains only the new HRR parameters (theta, beta, hnorm, mix) by distillation, and uploads gates + logs to the Hub as it goes.

Files

Training (from train_log.json)

The distillation loss is noisy step to step (it spikes on hard batches); the last-10 median is the number to compare.

Status and evidence

Gates are uploaded and load cleanly, and the swapped model runs — but passkey recall still fails, so this is not a usable model yet. What is verified: the attention state is constant (3.3 MB) regardless of context, and the swap is a pure distillation of the base model's own attention (no base weights changed).

The from-scratch evidence for the operator itself (softmax vs gated-HRR, matched pair) is in staccs/lecore-303m-rerun: at 303M / 1B tokens gated-HRR lands 41.5% worse in validation loss (4.5138 vs 3.1902; 6.5120 vs 4.6025 bits/byte). Cheap constant memory, a real quality price. Method write-ups, the tiny-twin benchmarks and the footprint measurements: staccs/lecore-hrr-gpt2.

How to use

import torch
from hrr_swa import convert, new_params   # code/hrr_swa.py
convert(model, 256)                    # swap attention for HRRSWAAttention, window 256
state = torch.load("hrr_gates.pt", map_location="cpu")
# load the trained theta / beta / hnorm / mix tensors into the swapped modules

License

Base model: swiss-ai/Apertus-70B-Instruct-2509. The base weights are untouched — only the HRR gate tensors here were trained.