Supra2.5-Pico

2.98M parameters · pretrained from scratch · 12.96 on the Open SLM Leaderboard Intelligence Index

Supra2.5-Pico is the smallest model in the Supra2.5 line: a decoder-only language model with 2,982,480 parameters, trained from scratch on 12B tokens of English text with a custom 4,104-token tokenizer. It is a base model for text completion. It has not been instruction-tuned, chat-tuned or aligned in any way.

At release it scores 12.96 on the Open SLM Leaderboard metric. That is the highest score of any model under 30M parameters on the leaderboard as of 22 October 2026 - 0.80 points above Supra2-Medium-Instruct (12.16, 25.4M) and 3.35 points clear of GPT-S3-3M (9.61), the best sub-3M entry.

It is also the most parameter-efficient model in its class that we measured: 4.35 Intelligence Index points per million parameters, against 3.23 for GPT-S3-3M and 1.78 for the nearest 10M-class entry listed below.

---

Performance

Zero-shot, acc_norm, full test sets, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script on the float32 weights in this repository.

The Index is (N(HS,25) + N(ARCavg,25) + N(PIQA,50) + 0.65·N(AM3,25)) / 3.65, where N(v,c) = 100·(v−c)/(100−c) and ARCavg is the mean of ARC-Easy and ARC-Challenge.

Against the sub-3M class

Figures for the other models are from the Open SLM Leaderboard or the respective model cards. They are shown for context only and were not re-measured here.

Against the sub-30M board

Best entries under 30M parameters, recomputed from the leaderboard's published figures on 22 October 2026:

Efficiency

---

Model details

Architecture

Tokenizer

A 4,104-token byte-level BPE: 4,096 merges plus eight length tokens <|len1|>…<|len8|>. Numbers are written with their length token and their digits in units-first order, on both the encode and the decode side; 62 becomes <|len2|> 2 6. Loading the tokenizer with trust_remote_code=True reproduces this behaviour automatically.

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2.5-Pico"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.float32
).eval()

prompt = "The water cycle begins when"
ids = tok(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=64, do_sample=False,
                         repetition_penalty=1.1, no_repeat_ngram_size=4)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required: SLMForCausalLM and the tokenizer class are defined in modeling_slm.py and tokenization_pascaline.py inside this repository.

Evaluate in float32. bf16 rounding moves logits enough to flip borderline multiple-choice answers.

---

Reproducing the evaluation

Everything needed is in eval/:

eval/reproduce_evaluation.py   runs the harness, ArithMark, and recomputes the index
eval/bencharithmark-3.py       official ArithMark-3 scorer (1,000 items, acc_norm)
eval/arithmark-3.jsonl         the 1,000 ArithMark-3 items

pip install -U lm-eval accelerate torch transformers datasets
cd eval
python reproduce_evaluation.py --model . --output results

Or by hand, which is what the numbers above come from:

python -m lm_eval --model hf \
  --model_args pretrained=SupraLabs/Supra2.5-Pico,trust_remote_code=True,dtype=float32 \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 8 --device cuda

python bencharithmark-3.py --model . --data-path arithmark-3.jsonl \
  --dtype float32 --device cuda --batch-size 8

batch_size 8 is deliberate: at 1,024 context, larger batches exhaust a 24GB card when the evaluation runs alongside anything else.

---

Intended use

Intended: research on small-scale pretraining, scaling behaviour and tokenizer design; a cheap starting point for fine-tuning; education; fast iteration on a single GPU.

Not intended: production or user-facing deployment, factual question answering, decision support, non-English text. At three million parameters the model confabulates freely, repeats itself on long generations, and its arithmetic is limited to short grade-school items.

Limitations

• English only, 1,024-token context, no instruction tuning, no safety tuning, no RLHF.
• Weak on general language modelling: it loses roughly 3 points of LAMBADA accuracy and 0.09 bits/byte on WikiText-2 to GPT-S3-3M, which saw more than twice as many tokens.
• Arithmetic ability is narrow. The ArithMark-3 score reflects the benchmark's task shape at this model size and should not be read as general mathematical reasoning.
• No chat template is provided. Applying one will not produce sensible output.

Citation

@misc{supra25pico,
  title  = {Supra2.5-Pico},
  author = {SupraLabs},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/SupraLabs/Supra2.5-Pico}}
}

License

Apache 2.0. See LICENSE.