Surjo-50M

Surjo-50M is a 53.8M-parameter hybrid recurrent language model developed by SurjoLabs. Built on the Gen-3 Surjo architecture, it couples Gated DeltaNet-2 (GDN-2) linear attention with full attention (XSA).

By executing a weight-tied recurrent core across 2 passes, the model reaches an effective computational depth of 18 layers while retaining the memory footprint and latency profile of a 10-layer physical model.

Instruction-Tuned Variant: For conversational use, instruction following, and a 2048-token context window, see Surjo-50M-SFT-Only.

Model Specifications

Attribute	Specification
Parameters	53.8M
Architecture	Hybrid GDN-2 + XSA (Subtractive Attention)
Physical Layers	10 (1 Prelude + 8 Recurrent Trunk + 1 Coda)
Effective Depth	18 Layers (2 Recurrent Passes)
Hidden Size	512
Intermediate Size	1536
Attention Configuration	GQA (8 Query heads, 4 KV heads; Head Dim: 64)
Linear Attention	GDN-2 (K: 64, V: 64, 8 V-Heads)
Context Length	1024 tokens pretraining (2048 max)
Vocabulary	32,768 (Custom BPE, tied embeddings)

Evaluation Results

Benchmark	Score
PIQA	63.11%
ARC-Easy	44.99%
ArithMark-3	37.90%
HellaSwag	31.37%
ARC-Challenge	25.94%
Int Index	16.40

Architecture Details

Surjo Gen-3 combines fast recurrent sequence processing with full-attention boundary layers:

Prelude (Layer 0): Standard attention with subtractive value projection (XSA) to anchor context.
Recurrent Core (Layers 1–8, tied 2× passes): Two interleaved groups consisting of 3× GDN-2 linear attention layers followed by 1× XSA full attention layer. GDN-2 uses chunked/fused recurrent kernels via Flash Linear Attention (FLA).
Coda (Layer 9): Terminal full-attention XSA layer.
Tied Head: The output projection shares weights with the 32,768-token embedding layer.

Pretraining

Dataset (20B tokens):
- 60% FinePhrase
- 20% DCLM
- 10% FineMath
- 10% Cornstack Python
Training Schedule: 20,000 steps (~1M tokens/step) at a sequence length of 1024 tokens.
Optimizer: Hybrid Muon (peak lr = 0.02) and AdamW (peak lr = 0.0036).

Usage

Requirements

pip install torch transformers accelerate
pip install -U git+https://github.com/fla-org/flash-linear-attention

Generation Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Surjo-50m"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The secret to scaling small language models is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & License

This project is released under the Apache 2.0 License.

@software{surjo50m2026,
  author = {SurjoLabs},
  title = {Surjo-50M: Hybrid Recurrent Language Model},
  year = {2026},
  url = {https://huggingface.co/SurjoLabs/Surjo-50m}
}