Byrne-86M-Base

🔭 Jet-Long context extension (native 4K → 10K)

This is the Jet-Long edition of Byrne-86M-Base. It extends the usable context from the native 4,096-token training window to 10,240 tokens with no fine-tuning and no change to short-context behaviour, by adding dynamic bifocal RoPE from Jet-Long (arXiv:2607.07740, NVIDIA).

What was applied

Jet-Long pairs a local window (w0 = 2048, classic RoPE) with a remote window whose position map aliases far-apart tokens back onto the pretrained rotation grid:

f(x) = floor(x / G),   G = max(1, ceil(L / 4096))

G adapts to the current sequence length L, so:

• L ≤ 4096 → G = 1 → f is the identity → the model is bit-for-bit the base model. (Verified: max |Δlogit| between Jet-Long on/off within the window is 0.000e+00.)
• L > 4096 → the remote window keeps every rotation in-distribution, so the model extrapolates instead of collapsing.

Implementation notes specific to this SpikeWhaleLM build:

• Only the decoupled RoPE partition (16 of 64 head dims) is aliased; the NoPE partition is untouched. Softmax attention (use_derf=False) — the standard Jet-Long merge applies.
• The remote view is realized by an on-the-fly correction rotation on the already-RoPE'd KV cache (RoPE composes additively), so the cache is never rewritten and decode is cheap.
• Enabled via config: use_jetlong=true, jetlong_w0=2048, jetlong_w_pretrained=4096, max_position_embeddings=10240. Set use_jetlong=false to recover the exact base model.
• The inclusion–exclusion / CuTe throughput kernel from the paper is not included (it targets 100K+ contexts on H100); at 86M params the bifocal attention is computed directly.

Measured (PG-19-style perplexity on held-out text, lower is better)

Beyond the training window the base model's perplexity blows up while Jet-Long stays flat — and long-context generation stays grammatical where the base model degrades into word-salad.

Usage

Jet-Long is on by default in this repo. Pass explicit position_ids so RoPE gets true absolute positions during cached decode:

import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-Base-JL", trust_remote_code=True)
ids = ...              # up to ~10,240 tokens
pos = torch.arange(ids.shape[1]).unsqueeze(0)
out = m(input_ids=ids, position_ids=pos, use_cache=True)   # prefill, then decode step-by-step

Method: Tang, Wang, Gu, Han, Cai — “Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE”, arXiv:2607.07740. Applied here zero-shot to SpikeWhaleLM; no weights were retrained.

The base model of the Byrne family (distilled step-4000 checkpoint) — a strong general base for continued pretraining / fine-tuning. A ~86M-parameter, from-scratch SpikeWhaleLM decoder (Multi-head Latent Attention, n-gram engram memory, hash-lookup layers, hyper-connections, HRM refinement, MTP) with a custom ChatML-aware tokenizer. Trained with Modal credits during the Small Models, Big Adventures Hackathon.

Related: main model → Byrne-86M

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)

Architecture

These models are built on SpikeWhaleLM, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components:

• Multi-head Latent Attention (MLA + XSA) — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a single KV head (multi-query attention), with QK-norm for stable logits.
• Engram n-gram memory — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream.
• Hash-lookup layers (×2) — multi-head content-addressable features alongside the token embeddings.
• Hyper-Connections — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add.
• HRM refinement — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head.
• Multi-Token Prediction (MTP) — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost).
• Feed-forward is dense (the block is MoE-capable, but MoE is disabled in this release).

JEPA vs HRM. The Byrne models are Non-JEPA: they are trained with HRM refinement only (use_hrm_refine=True, use_jepa=False). The sibling Escarda models add a JEPA (Joint-Embedding Predictive) auxiliary objective on top of HRM refinement.

Tokenizer

These models use SpikeTokenizer, a custom byte-level "length-max" (greedy longest-match) tokenizer with a 16,512-token vocabulary — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is ChatML-aware, with atomic special tokens for framing and reasoning/tool markers (<|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers) plus <bos>/<eos>/<pad>/<unk>. It ships as a PreTrainedTokenizer subclass (spike_tokenizer.py) and loads via AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

log-likelihood, acc_norm = byte-length-normalized).

ArithMark-2.0 (AxiomicLabs) — official metric is raw acc: 0.2732.

Language modeling: WikiText-2 byte_ppl (↓) 2.3753 · BLiMP (↑) 0.7356.

Citation

If you use this model, please cite:

@misc{byrne86mbase,
  title        = {Byrne-86M-Base: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M-Base}},
  note         = {Quazim0t0/Byrne-86M-Base}
}

Update: engram repair (behavior-preserving)

The n-gram Engram memory in the original weights was degenerate: with the frozen LSH compressor at init scale, every token hashed to bucket 0, so only one table row ever received gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.

Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram's hash now spreads across the full table and every bucket is independently trainable — so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.