DeepSeek-R1-Distill-Llama-70B-lecore-hrr — leCore gated-HRR attention gates (WORK IN PROGRESS)
Work in progress — gates were never uploaded for this model. This repo holds the training code (and log, where the run produced one), not a finished adapter. See the per-model note below.
What this is
A drop-in replacement for full attention in deepseek-ai/DeepSeek-R1-Distill-Llama-70B: the layer keeps a
sliding-window softmax (fixed window W=256) plus a gated HRR (holographic reduced
representation) recurrent state, so decode memory is O(1) in context length instead of a KV cache
that grows with it. code/hrr_swa.py holds the module (HRRSWAAttention(old, cfg, window=256)),
code/gated_scan_triton.py the Triton scan kernel, code/bigjob.py the training driver: it loads one
copy of the model, toggles the original attention as the teacher, trains only the new HRR
parameters (theta, beta, hnorm, mix) by distillation, and uploads gates + logs to the Hub as it goes.
Files
| file | size | what it is |
|---|---|---|
code/bigjob.py |
8.8 KB | training driver: teacher = original attention, student = hybrid HRR |
code/hrr_swa.py |
8.4 KB | the attention module (HRRSWAAttention, convert, new_params) |
code/gated_scan_triton.py |
5.0 KB | Triton gated scan (10x faster than the Python loop, 75 → 42 GB) |
Training (no log in this repo)
No train_log.json in this repo. code/bigjob.py is the driver: set MODEL, REPO, HF_TOKEN, WINDOW, SEQ, NTOK and run it on one GPU large enough to hold the bf16 base model plus the HRR state.
Status and evidence
Code only — the 70B run never uploaded gates. code/bigjob.py + code/hrr_swa.py are the exact driver/module used for the other 70B-scale runs (Llama-3.3-70B-Instruct-lecore-hrr, Qwen2.5-72B-Instruct-lecore-hrr, Apertus-70B-Instruct-2509-lecore-hrr).
The from-scratch evidence for the operator itself (softmax vs gated-HRR, matched pair) is in staccs/lecore-303m-rerun: at 303M / 1B tokens gated-HRR lands 41.5% worse in validation loss (4.5138 vs 3.1902; 6.5120 vs 4.6025 bits/byte). Cheap constant memory, a real quality price. Method write-ups, the tiny-twin benchmarks and the footprint measurements: staccs/lecore-hrr-gpt2.
How to use
No gates in this repo yet — code/bigjob.py is the whole pipeline: it loads the base model, calls convert(model, 256), trains the new HRR parameters against the original attention as teacher, and uploads hrr_gates.pt when it finishes.
License
Base model: deepseek-ai/DeepSeek-R1-Distill-Llama-70B. The base weights are untouched — only the HRR gate tensors here were trained.