Byrne-TriAtn-86M

A TriAttention-enabled variant of Byrne-86M. Same weights, same chat behavior — plus a training-free KV-cache compression path (TriAttention) that bounds attention memory at long context. This repo does not replace Byrne-86M; it is a separate, drop-in model.

For contexts up to the KV budget (default 2048 tokens), this model's output is byte-for-byte identical to Byrne-86M (verified). TriAttention only changes behavior once the cache would exceed the budget, where it caps memory instead of growing.

What was changed vs Byrne-86M

Nothing was retrained — the transformer weights are identical. The additions are inference-side:

• triattention.py (new) — a per-layer KV-cache compressor implementing TriAttention: it scores cached keys in the pre-RoPE space using this model's Q/K concentration centers (a trigonometric series over Q–K distance) plus a Q/K-norm signal, and keeps the top-scoring keys (with attention-sink + recent-window retention). Streaming, position-aware eviction.
• Baked calibration — the per-layer pre-RoPE Q/K centers, concentration (mean-resultant-length R ≈ 0.79), and key-norm statistics are stored as buffers in model.safetensors. Calibrated offline on 256 FineWeb-Edu documents (the family's pretraining distribution). No weight updates.
• Config flags (config.py) — use_triattention (default True), tri_kv_budget (default 2048), tri_local_window, tri_sink_tokens, tri_future_offsets.
• Dynamic RoPE (model_v2.py) — the rotary tables now extend on demand, so generation can exceed the original 4096-position cap.

Helper scripts included: calibrate_triattention.py, generate_triattention.py, compare_triattention.py, chat_eval_triattention.py.

Why / when it helps

KV cache grows linearly with sequence length. TriAttention caps it at tri_kv_budget, cutting memory at long context (and stabilizing throughput) with little quality loss. On an 86M model this is a memory tool, not a speed tool — the benefit appears when the cache would otherwise grow large.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-TriAtn-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-TriAtn-86M", trust_remote_code=True)
# TriAttention is ON by default (budget 2048). To tune or disable:
#   ...from_pretrained(..., tri_kv_budget=1024)      # more compression
#   ...from_pretrained(..., use_triattention=False)  # exact original behavior

Prompts use ChatML (<|im_start|>role\n…<|im_end|>), generation starting after a trailing <|im_start|>assistant\n. Note: when using TriAttention with compression active you must pass explicit absolute position_ids during generation (see generate_triattention.py), because a compressed cache can no longer infer position from its length.

Measured behavior (this model)

• Short chat (≤ budget): identical to Byrne-86M.
• Long context (ChatML, ~3.3k tokens): identical to full attention down to budget ≈ 1024 (~3× KV reduction) on the tested prompt; quality drifts below that. Default 2048 leaves a safe margin.

Limitations

• Adaptation of TriAttention to MLA / partial-RoPE / MQA; pooled per-head scores onto the shared KV head. Validate on your own workload before relying on aggressive budgets.
• 86M chat model: best on short, ChatML-formatted turns; long summarization is repetitive (a base-capacity limit, independent of compression).
• Custom SpikeTokenizer is not bf16-safe in the engram path; run in fp32/fp16.

Architecture, tokenizer, evaluation

Unchanged from Byrne-86M (SpikeWhaleLM, ~86M params, 16 layers, hidden 640, MLA + RoPE-16/NoPE-48 MQA, engram memory, hyper-connections, HRM refinement, Non-JEPA, ChatML SpikeTokenizer). See the base repo for the full benchmark table.

Citation

@misc{byrnetriatn86m,
  title        = {Byrne-TriAtn-86M: TriAttention KV-compression variant of Byrne-86M},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-TriAtn-86M}},
  note         = {TriAttention: arXiv:2604.04921}
}

Update: format-blended SFT on the engram-repaired base

This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: ') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:

MMLU acc 0.280->0.276, format 0.926->0.948; GSM8K '####' 0.420->0.790

Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.