Byrne-86M

Main Byrne-86M chat checkpoint (OPD v2). The one I point people at for general use. ~86M SpikeWhaleLM from scratch - MLA, n-gram engram, hash-lookup, hyper-connections, HRM refine, MTP. Custom ChatML-aware tokenizer.

Trained with Modal credits during the Small Models, Big Adventures Hackathon.

Related: base → Byrne-86M-Base

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)

Architecture

SpikeWhaleLM, ~86M, 16 layers, hidden 640, 4096 context, 16,512 vocab, tied embeddings.

• Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128); each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head (MQA); QK-norm.
• Engram n-gram memory - gated table, hashes local n-grams (up to trigrams) into 4,096 rows, mixes back into the residual.
• Hash-lookup layers (×2) - content-addressable features next to the token embeddings.
• Hyper-Connections - learned width-expanded residuals, Sinkhorn routing, instead of a plain add.
• HRM refinement - extra latent pass over hidden states before the output head.
• Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one next token. Training only.
• FFN is dense. The block can do MoE; MoE is off in this release.

JEPA vs HRM. Byrne is Non-JEPA: HRM refine only (use_hrm_refine=True, use_jepa=False). Escarda adds JEPA on top of HRM.

Tokenizer

SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab. Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware. Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>. PreTrainedTokenizer in spike_tokenizer.py. Load with AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

Zero-shot multiple-choice, continuation log-likelihood (acc_norm = byte-length-normalized).

ArithMark-2.0 (AxiomicLabs)

• official metric is raw acc: 0.3096.

Language modeling: WikiText-2 byte_ppl (↓) 2.6839 · BLiMP (↑) 0.7033.

Citation

If you use this model, please cite:

@misc{byrne86m,
  title        = {Byrne-86M: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M}},
  note         = {Quazim0t0/Byrne-86M}
}

Update: format-blended SFT on the engram-repaired base

This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: ') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:

MMLU acc 0.260->0.254, format 0.918->0.951; GSM8K '####' 0.420->0.815

Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.