Byrne-86M-Base

Base of the Byrne family. Distilled step-4000 checkpoint. ~86M SpikeWhaleLM from scratch - MLA, n-gram engram, hash-lookup, hyper-connections, HRM refine, MTP. Custom ChatML-aware tokenizer. I use this as a general base to keep pretraining / SFT.

Trained with Modal credits during the Small Models, Big Adventures Hackathon.

Related: chat → Byrne-86M

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)

Architecture

SpikeWhaleLM, ~86M, 16 layers, hidden 640, 4096 context, 16,512 vocab, tied embeddings.

• Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128); each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head (MQA); QK-norm.
• Engram n-gram memory - gated table, hashes local n-grams (up to trigrams) into 4,096 rows, mixes back into the residual.
• Hash-lookup layers (×2) - content-addressable features next to the token embeddings.
• Hyper-Connections - learned width-expanded residuals, Sinkhorn routing, instead of a plain add.
• HRM refinement - extra latent pass over hidden states before the output head.
• Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one next token. Training only.
• FFN is dense. The block can do MoE; MoE is off in this release.

JEPA vs HRM. Byrne is Non-JEPA: HRM refine only (use_hrm_refine=True, use_jepa=False). Escarda adds JEPA on top of HRM.

Tokenizer

SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab. Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware. Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>. PreTrainedTokenizer in spike_tokenizer.py. Load with AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

Zero-shot multiple-choice, continuation log-likelihood (acc_norm = byte-length-normalized).

ArithMark-2.0 (AxiomicLabs)

• official metric is raw acc: 0.2732.

Language modeling: WikiText-2 byte_ppl (↓) 2.3753 · BLiMP (↑) 0.7356.

Citation

If you use this model, please cite:

@misc{byrne86mbase,
  title        = {Byrne-86M-Base: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M-Base}},
  note         = {Quazim0t0/Byrne-86M-Base}
}

Update: engram repair (behavior-preserving)

The n-gram Engram in the original weights was degenerate: frozen LSH compressor at init scale hashed every token to bucket 0, so only one table row ever got gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.

Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram hash now spreads across the full table and every bucket is independently trainable - so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.