Sign up

SurjoLabs/Surjo-100m

🤗 Hugging Face sourcetext-generationapache-2.098M params391 MBsafetensors✓ HF checksums availableupdated today
No torrent yet →
SurjoLabs

Surjo-100m

Older Sister. Higher Standards.

Surjo-100m is a 97.7M-parameter hybrid recurrent language model developed by SurjoLabs. Built on the Gen-3 Surjo architecture, it couples Gated DeltaNet-2 (GDN-2) linear attention with full attention (XSA).

By executing a weight-tied recurrent core across 2 passes, the model reaches an effective computational depth of 34 layers while retaining the memory footprint and latency profile of an 18-layer physical model.

Model Specifications

Attribute Specification
Parameters 97.7M
Architecture Hybrid GDN-2 + XSA (Subtractive Attention)
Physical Layers 18 (1 Prelude + 16 Recurrent Trunk + 1 Coda)
Effective Depth 34 Layers (2 Recurrent Passes)
Hidden Size 576
Intermediate Size 1536
Attention Configuration GQA (9 Query heads, 3 KV heads; Head Dim: 64)
Linear Attention GDN-2 (K: 64, V: 64, 9 V-Heads)
Context Length 1024 tokens pretraining (2048 max)
Vocabulary 32,768 (Custom BPE, tied embeddings)

Evaluation Results

Benchmark Score
PIQA 63.87%
ARC-Easy 47.64%
ArithMark-3 38.90%
HellaSwag 35.05%
ARC-Challenge 25.85%
Int Index 18.86

Architecture Details

Surjo Gen-3 combines fast recurrent sequence processing with full-attention boundary layers:

  • Prelude (Layer 0): Standard attention with subtractive value projection (XSA) to anchor context.
  • Recurrent Core (Layers 1–16, tied 2× passes): Four interleaved groups consisting of 3× GDN-2 linear attention layers followed by 1× XSA full attention layer. GDN-2 uses chunked/fused recurrent kernels via Flash Linear Attention (FLA).
  • Coda (Layer 17): Terminal full-attention XSA layer.
  • Tied Head: The output projection shares weights with the 32,768-token embedding layer.

Pretraining

  • Dataset (20B tokens):
    • 60% FinePhrase
    • 20% DCLM
    • 10% FineMath
    • 10% Cornstack Python
  • Training Schedule: 20,000 steps (~1M tokens/step) at a sequence length of 1024 tokens.
  • Optimizer: Hybrid Muon (peak lr = 0.02) and AdamW (peak lr = 0.0036).

Usage

Requirements

pip install torch transformers accelerate
pip install -U git+https://github.com/fla-org/flash-linear-attention

Generation Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Surjo-100m"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The secret to scaling small language models is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & License

This project is released under the Apache 2.0 License.

@software{surjo100m2026,
  author = {SurjoLabs},
  title = {Surjo-100m: Hybrid Recurrent Language Model},
  year = {2026},
  url = {https://huggingface.co/SurjoLabs/Surjo-100m}
}