Escarda-86M-Base

~86M from-scratch decoder. Base sibling of Quazim0t0/Escarda-86M (the chat one). Same SpikeWhaleLM stack: MLA, n-gram engram, hash-lookup, hyper-connections, HRM refine, JEPA / multi-token-prediction. Same ChatML-aware tokenizer.

JEPA-distilled base. Use it to keep pretraining / SFT. Not a chat assistant.

Related: chat → Quazim0t0/Escarda-86M · demo → Escarda-86M-Chat Space

Trained on Modal credits for the Small Models, Big Adventures Hackathon.

---

Model summary

Full architecture writeup is on the chat card.

---

Architecture

SpikeWhaleLM, ~86M, 16 layers, hidden 640, 4096 context, 16,512 vocab, tied embeddings.

• Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128); each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head (MQA); QK-norm.
• Engram n-gram memory - gated table, hashes local n-grams (up to trigrams) into 4,096 rows, mixes back into the residual.
• Hash-lookup layers (×2) - content-addressable features next to the token embeddings.
• Hyper-Connections - learned width-expanded residuals, Sinkhorn routing, instead of a plain add.
• HRM refinement - extra latent pass over hidden states before the output head.
• Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one next token. Training only.
• FFN is dense. The block can do MoE; MoE is off in this release.

JEPA vs HRM. Escarda trains both HRM refine and JEPA (use_hrm_refine=True, use_jepa=True). JEPA predicts future latents during training. Byrne drops JEPA and keeps HRM only.

Tokenizer

SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab. Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware. Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>. PreTrainedTokenizer in spike_tokenizer.py. Load with AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

Zero-shot multiple-choice, continuation log-likelihood on each task's val/test splits. byte_ppl is exp(sum_NLL_nats / total_UTF8_bytes) on WikiText-2 test (tokenizer-independent). BLiMP is fraction of minimal pairs with logprob(good) > logprob(bad) (12 paradigms × 150). Stderr is binomial sqrt(p(1-p)/n).

Language modeling

Multiple-choice suite

ArithMark-2.0 (AxiomicLabs)

n = 2,500 · chance = 0.25.

Distilled base. Lowest byte-perplexity in the Escarda family, weaker downstream. PPL is not a capability ranking. Chat behaviour: Escarda-86M. This one if you want a low-loss base to keep training.

---

Training & token budget

• Tokens: ~20B from-scratch SpikeWhale (~28k steps). This checkpoint is a JEPA-distilled snapshot of that base.
• Token/param: ~233 (20B / 85.7M). About 11-12× Chinchilla's ~20-tokens/param. Over-trained small model on purpose. Inference is the trade.

Fitting Chinchilla's data term to this run's pretrain loss:

L(D) ≈ 2.611 + 77,715 · D^(-0.537) (nats/token, R² = 0.92)

From that:

• Compute-optimal for this 86M ≈ 4.3B → 20B is ~4.6× past compute-optimal.
• Diminishing-returns knee ≈ 22.5B (where +1B buys < 0.005 nats). 20B lands right there.
• Parameter-bound, not data-bound at 20B: capacity term (~0.82 nats) beats the data term (~0.54). Extra tokens do little. Doubling to 40B is projected ~0.07 nats lower loss (~7% PPL) with basically no downstream gain. Next lever is more params, not more tokens. Same reason this distilled base has the lowest PPL in the family without the best downstream scores - it is already on the data-term floor.

Caveats: single-size fit (irreducible loss + capacity floor folded into one constant). Cosine-LR decay inflates the fitted exponent, so treat β as an upper bound. Token counts are anchored to ~20B and scale linearly if that figure is off.

---

Usage

Custom architecture. trust_remote_code=True. Modeling code is in the repo (auto_map):

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Quazim0t0/Escarda-86M-Base", trust_remote_code=True)

Tokenizer is SpikeTokenizer (tokenizer.json, algorithm: length-max). Load it with spike_tokenizer.py from the project, not AutoTokenizer.

Acknowledgements

Modal credits, Small Models, Big Adventures Hackathon. Released as a base to build on.

Citation

If you use this model, please cite:

@misc{escarda86mbase,
  title        = {Escarda-86M-Base: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Escarda-86M-Base}},
  note         = {Quazim0t0/Escarda-86M-Base}
}

Escarda vs Byrne - vision family comparison

Byrne = HRM refine. Escarda = Byrne + JEPA on the vision encoder and the LM trunk. Auxiliary only. Zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags. Byrne-Docling is a bit more complete on the hardest samples (closes </formula>, includes the <code> wrapper), which matches the slightly higher teacher-alignment. Escarda-Docling is structurally on par and has the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1-3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M

Update: engram repair (behavior-preserving)

The n-gram Engram in the original weights was degenerate: frozen LSH compressor at init scale hashed every token to bucket 0, so only one table row ever got gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.

Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram hash now spreads across the full table and every bucket is independently trainable - so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.