Byrne-VLM (131M)

Model will be ungated for open download once I am done with the base.

~130.9M VLM, all SpikeWhale / Byrne: 39M vision encoder from scratch, a small connector, ~90M Byrne LM. LLaVA-style, trained by distillation. Research / architecture artifact, not a SOTA captioner (see Honest evaluation).

image → Byrne-VE (39.3M, frozen) → 196 patch tokens (512-d)
      → Connector MLP (512→640, 1.18M)
      → spliced at <image> placeholders into Byrne-LM (90M + Family-LoRA)
      → caption

Components

How it was trained

1 · Vision encoder (Byrne-VE). Frozen facebook/dinov2-base teacher (cosine on CLS + patch grid), then teacher-free DINO-style EMA self-distillation. ~88% of DINOv2 k-NN at ~45% of the params. HRM gate alive (tanh ≈ 0.35).

2 · Connector grounding (10 streamed rounds). Connector maps vision into the LM. Streamed image-caption pairs: CC3M, CC12M, Conceptual-Captions-12M, LLaVA-ReCap-558K/118K/CC3M, TextCaps, Flickr8k, LLaVA-NeXT-Data. Vision encoder

• LM frozen.

3 · Style fine-tune. Connector on ~25k images captioned by HuggingFaceTB/SmolVLM-256M-Instruct - sequence-level distillation (teacher and SpikeTokenizer have different vocabs, so we distill the generated caption text, not logits).

4 · Stage-2 Family-LoRA. Family-LoRA (HRM + MoE-SwiGLU) on the LM decoder, trained with the connector. Base LM frozen. Zero-init adapter wakes (gates climb off zero, MoE routing learns, no collapse).

Honest evaluation

This is a demo of SpikeWhale as a VLM, not a competitive captioner. 500 COCO-val: CIDEr ≈ 0.06 / BLEU-4 ≈ 4.8 - far below usable captioners (the teacher, SmolVLM-256M, is much stronger). It gets coarse scene gist and clear subjects (e.g. lion, tower) but not specific, reference-grade captions. Capacity-and-scale (90M from-scratch LM, 39M encoder, ~25k caption pairs), not a wiring bug. Examples:

Usage

import torch
from generate import load_vlm, caption
from spike_tokenizer import SpikeTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
tok = SpikeTokenizer(vocab_file="tokenizer.json")
vlm = load_vlm("weights/byrne_vlm.pt", "lm", "weights/byrne_ve.pt", device)
print(caption(vlm, tok, "photo.jpg", device))

AnyRes tiling (higher effective resolution for dense images / documents) via load_vlm(..., anyres_grid=(2,2)) - fits 4096 context (980 image tokens) and needs a matching fine-tune.

Files

weights/byrne_vlm.pt (connector + Family-LoRA) · weights/byrne_ve.pt (vision encoder) · lm/ (Byrne base LM) · model code (vlm_model.py, modeling_byrne_embed.py, family_lora.py, anyres.py, …).

Citation

@misc{byrne_vlm_2026,
  title  = {Byrne-VLM: A Tiny From-Scratch SpikeWhale Vision-Language Model},
  author = {Quazim0t0},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-VLM-131M}}
}

License

Apache-2.0.

Escarda vs Byrne - vision family comparison

Byrne = HRM refine. Escarda = Byrne + JEPA on the vision encoder and the LM trunk. Auxiliary only. Zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags. Byrne-Docling is a bit more complete on the hardest samples (closes </formula>, includes the <code> wrapper), consistent with its slightly higher teacher-alignment. Escarda-Docling is structurally on par and has the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1-3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M

v2 update (448px + repaired engram + terse captions)

Upgraded: 448px vision encoder (was 224px), engram-repaired base LM (n-gram memory was degenerate - all tokens hashed to bucket 0 - now repaired bit-exact and populated), retrained on terse COCO-style captions.

COCO CIDEr (val, n=500): 0.061 -> 0.140 (2.3x). Intermediate verbose-style v2 scored 0.029. Switching training captions from "In this image we can see..." to terse COCO-style is what unlocked the gain.

Use this as a base - keep training on it

Meant as a base VLM you keep training, not a finished end-model. Ships with the vision encoder (weights/byrne_ve.pt, 448px), engram-repaired base LM (lm/), and a trained caption LoRA + connector (weights/byrne_vlm.pt). Fine-tune the LoRA on your own images/captions (or a new task) from here.

Two scripts: train_connector.py (stage 1, vision→LM grounding) and train_lora.py (stage 2, Family-LoRA that does the actual task).

Data format

Directory with captions.jsonl and images/:

my_data/
  captions.jsonl        # one JSON object per line
  images/
    0001.jpg
    0002.jpg

Each line of captions.jsonl:

{"image": "0001.jpg", "caption": "A cat sitting on a windowsill."}

Keep captions terse and COCO-style - verbose "In this image we can see..." hurts CIDEr badly (it cost this model 0.140 → 0.029 in an ablation).

A. Continue this model's LoRA (recommended)

Picks up the existing caption LoRA + connector and keeps going:

pip install -r requirements.txt

python train_lora.py --lora-type family \
    --lm-dir lm \
    --vision-ckpt weights/byrne_ve.pt \
    --connector   weights/byrne_vlm.pt \
    --resume-vlm  weights/byrne_vlm.pt \
    --data-dir my_data \
    --unfreeze-engram \
    --steps 4000 --batch-size 4 --lr 2e-4 \
    --save-every 1000 --out my_ckpt

--unfreeze-engram keeps the n-gram Engram trainable (repaired so every bucket is reachable - leave this on so the memory keeps populating). Result: my_ckpt/vlm_stage2_last.pt.

B. Train a fresh LoRA on your own task/style

Reuse the grounding (connector), start LoRA from scratch - different caption style or a new task. Drop --resume-vlm:

python train_lora.py --lora-type family \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --connector weights/byrne_vlm.pt \
    --data-dir my_data --unfreeze-engram \
    --steps 8000 --batch-size 4 --save-every 2000 --out my_ckpt

C. (Optional) Retrain the connector first

Only if you change the vision encoder or want fresh grounding:

python train_connector.py --data-dir my_data \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --steps 6000 --batch-size 4 --save-every 2000 --out my_ckpt
# then point train_lora.py --connector my_ckpt/connector_last.pt

Run your fine-tuned model

python generate.py --image my_data/images/0001.jpg \
    --ckpt my_ckpt/vlm_stage2_last.pt \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --tokenizer tokenizer.json --max-new 40 --repetition-penalty 1.2

Notes

• Document images / DocTags? Add --doctags --tokenizer tokenizer_doctags.json --letterbox and use structured-DocTags data - see Byrne-Docling for that path.
• Batch size / VRAM: batch 4 at 448px fits ~8-10 GB. Drop to 2 if you OOM.
• What trains: Family-LoRA (HRM + MoE-SwiGLU adapter, ~5M) + connector + (with --unfreeze-engram) the Engram tables. Base LM stays frozen otherwise.