Holo LLM — HRR Attention Conversions
Archive / conversion-track repo. Drop-in HRR attention experiments on Qwen2.5 (swap softmax for a fixed-size holographic state).
Live Status (2026-08-13)
The conversion-track table below is historical. Those “converting / queued / QLoRA” rows are not live jobs. Do not treat this card as the product.
Current product / live jobs (other repos / boxes):
| What | Where |
|---|---|
| DeepSeek-V4-Flash in-weight HRR overlay (memory 5/5 vs 0/5, KV spill needle) | staccs/lecore-deepseek-v4-flash-hrr |
| Qwen3.5-9B Unicron assimilate prove | staccs/lecore-qwen35-9b-assimilated |
| GPT-2 HRR vs softmax code + small ckpts | staccs/lecore-hrr-gpt2 |
| Qwen3.5-397B Galvatron | Vast 47593358 — not uploaded |
| OLMo 2 from-scratch HRR (original mix) | Vast 47603357 8×A100 — no loss yet, not this repo |
Historical conversion track (stale)
| Model | Status as originally written |
|---|---|
| Qwen2.5-0.5B-Instruct → HRR | Converted, fine-tuning (old) |
| Qwen2.5-7B-Instruct → HRR | Converting (old) |
| Qwen2.5-14B-Instruct → HRR | Queued (old) |
| Qwen2.5-32B-Instruct → HRR | QLoRA fine-tuning (old) |
| Qwen2.5-72B-Instruct → HRR | Queued (old) |
This repo still holds conversion scripts and some Qwen2.5-1.5B / gated-scan checkpoints. That is the artifact. It is not a hosted 72B.
How it was supposed to work
Each attention module is replaced with HRR attention — a gated holographic recurrence that stores context in a fixed-size superposed state. Conversion copies Q/K/V/O projection weights and adds learned per-frequency forgetting gates.
from transformers import AutoModelForCausalLM, AutoTokenizer
# only if the subfolder actually exists in this repo
model = AutoModelForCausalLM.from_pretrained("staccs/holo-llm-hrr-attention", subfolder="qwen0.5b_hrr")
See also
- lecore-deepseek-v4-flash-hrr — current Flash overlay
- lecore-hrr-gpt2 — from-scratch GPT-2 HRR proofs
Files in this repo
| file | size | what it is |
|---|---|---|
cloud/ckpt_Qwen2.5-1.5B_hrr_s1b.pt |
308.7 MB | support file |
data/enwik8 |
100.0 MB | support file |
checkpoints/ckpt_qwen_hrr_stage1b.pt |
88.3 MB | support file |
checkpoints/ckpt_softmax_T1024.pt |
49.6 MB | support file |
checkpoints/ckpt_gated_T1024_a0.9.pt |
46.1 MB | support file |
checkpoints/ckpt_qwen_hrr_stage1.pt |
82.6 KB | support file |
model.py |
14.2 KB | support file |
RESULTS.md |
10.7 KB | support file |