Supra2.6 Pico
2,972,496 parameters · 1,024 context · 4,096-token vocabulary · hybrid GDN-2 / gated-GQA
Apology for the accidental benchmaxing issues with 2.5 Pico.
Specifications
| Parameters (unique) | 2,972,496 |
| Parameters (as stored) | 3,562,320 tensors — the 589,824-parameter embedding is tied to the LM head and counted twice on disk |
| Non-embedding parameters | 2,382,672 |
| Layers | 8 |
| Layer pattern | [GDN-2, GDN-2, GDN-2, gated GQA] × 2 |
| d_model | 144 |
| FFN hidden | 352 (SwiGLU) |
| GDN-2 heads | 3 per layer, conv kernel 4, channel-wise erase/write + per-channel log-decay |
| GQA | 3 query heads / 1 KV head (head_dim 48), RoPE on 24 dims, q/k RMSNorm, sigmoid output gate |
| Attention residuals | learned softmax over all preceding layer outputs, one query per layer |
| Context length | 1,024 tokens |
| Vocabulary | 4,096 (BPE, trained on the training mixture) |
| Embeddings | tied input/output |
| Norm | pre-RMSNorm |
| Dtype | fp32 weights, trained in bf16 autocast |
| Weight file | model.pt — 11.9 MB |
Architecture
Eight blocks in two identical groups of four. Each group is three gated-delta-net (GDN-2) blocks followed by one gated grouped-query attention block, so 6 of 8 layers are linear-time recurrence and 2 carry global attention. GDN-2 uses the channel-wise erase/write recurrence
S_t = (I − k_t (b_t ⊙ k_t)ᵀ) Diag(exp(g_t)) S_{t−1} + k_t (w_t ⊙ v_t)ᵀ
implemented with the fla chunked kernel. Attention-residual mixing lets every layer read a learned softmax over the outputs of all previous layers.
Training recipe
| Tokens trained | 9,998,172,160 |
| Tokens per step | 262,144 (batch 256 × 1,024) |
| Steps | 38,140 |
| Optimizer | Muon (2-D matrices, lr 0.02, momentum 0.95, 5 Newton–Schulz steps) + AdamW (embeddings/norms, lr 6e-3, β 0.9/0.95) |
| LR schedule | linear warmup 10,000 steps, peak lr 0.02 held thereafter |
| Gradient clipping | 1.0 |
| Epochs per source | ≤ 1 (each source packed once, sampled proportionally) |
| Precision | bf16 autocast, fp32 master weights |
| Seed | 0 |
| Hardware | 1× RTX Pro 6000 Blackwell, 20 vCPU |
| Stack | PyTorch 2.x, fla-core 0.6.0 (chunked GDN-2 kernel), tokenizers |
Mixture (sampling shares; tokens sampled over the 9.998B run):
| source | share | tokens |
|---|---|---|
| ClimbMix | 36.3% | 3.63B |
| Cosmopedia v2 | 19.5% | 1.95B |
| Ultra-FineWeb L2 HQ | 19.5% | 1.95B |
| TXT360-v2 web-high-medium | 13.6% | 1.36B |
| open-web-math | 5.1% | 0.51B |
| OpenMathInstruct-2 | 4.8% | 0.48B |
| NuminaMath-CoT | 0.53% | 0.053B |
| TinyStories | 0.48% | 0.048B |
| OpenStax | 0.10% | 0.010B |
| orca-math-word-problems | 0.10% | 0.010B |
| GSM8K | 0.004% | 0.0004B |
Benchmarks
Full evaluation sets, log-likelihood scoring with length normalisation (acc_norm):
| benchmark | score | items |
|---|---|---|
| HellaSwag | 26.40 | 10,042 |
| ARC-Easy | 33.54 | 2,376 |
| ARC-Challenge | 23.21 | 1,172 |
| PIQA | 56.09 | 1,838 |
| Arithmark-3 | 38.10 | 1,000 |
| Index | 8.192 | — |
Index = ( N(HellaSwag, 25) + N(ARC mean, 25) + N(PIQA, 50) + 0.65 · N(Arithmark-3, 25) ) / 3.65, with N(v, c) = 100 · (v − c) / (100 − c).
Raw acc for the same checkpoint: HellaSwag 26.49 · ARC-Easy 33.50 · ARC-Challenge 17.92 · PIQA 56.91 · Arithmark-3 38.10.
Usage
model_h.py uses the Triton fla kernel; model_h_local.py (+ gdn2_ref.py) is the same model in pure PyTorch for CPU-only machines.
import torch
from tokenizers import Tokenizer
import model_h # or: import model_h_local as model_h
net = model_h.ModelH(vocab=4096, d=144, inter=352, n_gdn=6, n_gqa=2, ctx=1024)
net.load_state_dict(torch.load("model.pt", map_location="cpu", weights_only=False)["model"])
net.eval()
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("The capital of France is").ids])
logits = net(ids)[0]
model.safetensors stores the tied LM head once (155 tensors, 2,972,496 parameters); load it with strict=False after building the model, or use model.pt which carries both tied entries.