JugnuLM-110M-R1 (value residuals)

What "R1" means. This is rung 1 of the JugnuLM phase-2 ablation ladder. We take our honest 110M baseline — JugnuLM-110M (rung R0) — and add one architectural change, value residuals (ResFormer), holding everything else fixed (same data, tokens, schedule, batch, seed policy). The ladder keeps a lever only if it beats the previous rung. R1 does, so it's kept.

What value residuals are

Each attention layer's value gains a learned-gated residual from the first layer's value:

v_i = v_proj_i(x) + λ_i · v₀ (v₀ = layer-0 value; λ_i learned per layer)

This gives deep layers direct access to early-layer content (Zhou et al., Value Residual Learning, ACL 2025 — arXiv:2410.17897). The 22 learned gates converge to non-trivial values (range −2.5 to +3.2), i.e. the path is genuinely used.

Result vs the R0 baseline

Value residuals lift ARC-Easy by +2.2 at a BLiMP tie (SE ≈ 0.14) and a slightly lower perplexity — recovering part of the reasoning gap to the 125–143M leaderboard models, at negligible parameter cost. Evaluated with EleutherAI lm-evaluation-harness.

⚠️ Loading — requires trust_remote_code

Value residuals are a custom attention pathway, so this model must be loaded with the included modeling file. Loading it as a stock Qwen3 silently drops the residual and understates quality.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R1")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R1", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))

Architecture & training

Identical to R0 except the value-residual pathway: Qwen3-arch, 23 layers × 576, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152), z-loss 1e-4. Trained from scratch on ~8B tokens of FineWeb-Edu (sample-10BT), cosine LR 1.5e-3→1.5e-4, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell.

Intended use & limitations

Research base model (not instruction-tuned). Fluent English continuation and reasonable recall for its size; limited multi-step reasoning, occasional repetition, English-only. Not for production.

Links

• Baseline: altslate/JugnuLM-110M
• Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu

Paper

This model is described in Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M (arXiv:2609.14715 · HF Papers).

@misc{rajput2026jugnu,
  title         = {Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M},
  author        = {Rajput, Dushyant and Chauhan, Nirdesh and Kosaraju, Siddharth},
  year          = {2026},
  eprint        = {2609.14715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.14715}
}