Escarda-Docling-126M

~126M document-understanding VLM. Page image in, structured DocTags out (OTSL tables, <chart>/<formula>/<code>, <loc_*> boxes, language tags). Document specialization of the Escarda family.

Escarda = Byrne plus JEPA next to HRM (not instead of it). Everywhere Byrne has HRM refine, Escarda adds a JEPA head: from a token, predict a neighbouring token's representation (stop-gradient target, 1-cosine loss). Auxiliary only. Training only. Zero cost at inference.

Architecture

How it was built (identical recipe to Byrne-Docling)

1. VE distillation - 50k steps from a frozen DINOv2-base teacher (448px), HRM + JEPA aux.
2. VE self-distillation - 20k steps DINO-style (EMA teacher, cosine prototypes), no collapse (teacher peak-prob stayed ~0.15).
3. Connector - 6k steps aligning Escarda-VE -> Escarda-LM on document images (letterbox).
4. DocTags stage-2 - 8k steps Family-LoRA (HRM+MoE) with 140 atomic DocTags tokens added to the tokenizer + extended LM embeddings, letterbox preprocessing (whole page visible), on 6k ground-truth SmolDocling pairs (SynthChart/Formula/Code).

Escarda vs Byrne - head to head

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

Docling (same 3 held-out doc images, atomic DocTags):

Pros / cons

Byrne (HRM only)

• • Marginally higher teacher-alignment; slightly cleaner/more complete DocTags on these samples (closes formula, includes <code> wrapper).
• • All encoder capacity spent on distillation fidelity.
• • Pure supervised-mimicry representation; no self-supervised spatial objective.

Escarda (HRM + JEPA)

• • Adds a self-supervised neighbour-prediction (JEPA) objective in both the encoder and LM trunk -> richer spatial/representation structure, at zero inference cost.
• • Structurally sound DocTags, essentially on par with Byrne.
• • Trades ~1-3% teacher-alignment (JEPA diverts a little capacity from pure mimicry); slightly less complete on the hardest samples at this scale.

Bottom line: on raw DocTags quality the two are comparable, with Byrne holding a small edge on completeness; Escarda's value is the added JEPA representation-learning trait for free at inference. Both are the same size class.

Usage

pip install -r requirements.txt
python generate.py --image examples/chart.png --ckpt weights/escarda_docling.pt \
  --lm-dir lm --vision-ckpt weights/escarda_ve.pt --tokenizer tokenizer_doctags.json \
  --letterbox --max-new 256 --repetition-penalty 1.2 --no-repeat-ngram 3

Related

• Escarda-86M-Base - the JEPA+HRM base LM
• Byrne-Docling-131M - the HRM-only sibling
• Byrne-VLM-131M

Weights are bundled here for a self-contained release. Auxiliary JEPA adds no inference cost.

Engram repair (update)

The n-gram Engram memory in this checkpoint was degenerate: every token hashed to bucket 0, so only 1 of 4096 rows ever trained and the memory was a single constant (no further training could fix it). This version applies a behavior-preserving repair (rescale the frozen hash compressor + broadcast bucket 0) - outputs are unchanged - then a short distill with the Engram tables unfrozen so the now-reachable buckets populate.

Verified: hash bucket usage went from 1/4096 -> 4096/4096, and the Engram tables are now a genuinely populated n-gram memory instead of a constant. DocTags output remains valid.