100M-exp

A 98M-parameter base language model, trained from scratch on 21B tokens.

100M-exp is an experiment in small-model design: a hybrid sliding-window / full-attention transformer with a few additions meant to help at this scale. It was trained for 40k steps as a branch of a longer run, ending with a learning-rate cooldown. It scores 22.47 on the Open SLM Leaderboard's Intelligence Index (self-evaluated).

---

At a glance

---

Benchmarks

All scores are zero-shot, acc_norm, measured with our evaluation script that mirrors the leaderboard's Intelligence Index.

Intelligence Index: 22.47

Self-evaluated; the leaderboard's own run may differ by a few tenths.

How the Index is built

The Index measures how far above random guessing a model is, as a share of the gap between guessing and perfect, averaged across tasks:

In other words, 100M-exp covers about 22% of the distance between random guessing and a perfect score.

---

More benchmarks

Beyond the leaderboard tasks: zero-shot, lm-evaluation-harness, same model.

---

Architecture

100M-exp is a pre-norm decoder-only transformer with a hybrid attention layout: most layers only look at a local window, and two layers see everything.

Inside every block:

What each piece does, in plain terms:

• Sliding-window + full attention. 18 of 20 layers attend only to the previous 512 tokens, which is cheap and good at local grammar and phrasing. Two full-attention layers (7 and 15) connect the whole 8k context, so long-range information still flows.
• Attention sinks. Each head has a learned "nowhere" slot it can attend to, so it doesn't have to dump attention on real tokens when nothing is relevant (gpt-oss style).
• HoPE positions. A RoPE variant: rotary dimensions slower than one turn per 8k tokens are left un-rotated, giving the model some position-free channels for pure content matching.
• Short causal convolution. A tiny 4-token convolution on queries, keys and values lets each token blend in its immediate neighbours before attention, which tends to help small models.
• Value residual (ResFormer). Every block can mix in the first block's values, keeping early token information easy to reach in deep layers.
• Document masking. Documents packed into the same 8k sequence can never attend to each other, so the model never learns false connections between unrelated texts.
• GQA 9:3. Three key-value heads shared across nine query heads, which keeps attention lean.

---

Training

Data mix

A broad, quality-filtered web mix, an educational slice, long documents for the 8k context, and a math slice. Every training batch draws from every source.

Recipe

Schedule

100M-exp is a cooldown branch: it starts from the step-30k checkpoint of a longer 100k-step run and decays the learning rate to zero over 10k steps. Most of the final gain came from the cooldown:

---

Usage

The model uses custom code, so trust_remote_code=True is required.

Requirements: PyTorch 2.5 or newer (attention runs on FlexAttention); tested with transformers 5.14. On an Ampere or newer GPU (compute capability 8.0+) with triton, attention is compiled; on CPU, older GPUs (T4, V100), or if compiling fails, it falls back to chunked attention: the same results, with sliding-window layers scoring only the 512 keys each query can see, so memory grows with length instead of length squared (set SWA_NO_COMPILE=1 to force it). GPUs without bf16 (T4, V100) and CPUs: load with dtype=torch.float32. Generation uses a KV cache. Batches must not be padded: generate one prompt at a time, or batch only prompts of equal length.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/100M-exp"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()

prompt = "The process of photosynthesis"
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

---

Limitations

• A base model. It continues text; it was not trained to follow instructions or chat.
• Small. At 98M parameters it makes factual mistakes, loses the thread in long generations, and is weak at multi-step reasoning. Treat it as a research artifact, not a source of truth.
• Mostly English, reflecting its training data.
• Inherits web-data biases. The training mix is filtered web text, so outputs can reflect the biases and errors in that data.

---

Acknowledgements

100M-exp combines published ideas; none of the components below are new here.

Architecture

• Sliding-window and local/global attention: Longformer, Mistral 7B
• Attention sinks: Efficient Streaming Language Models with Attention Sinks; learned per-head sinks as in gpt-oss
• Rotary positions: RoFormer; HoPE
• Short convolution on queries/keys/values: Primer
• Value residual: Value Residual Learning (ResFormer)
• Grouped-query attention: GQA
• SwiGLU: GLU Variants Improve Transformer; RMSNorm: Root Mean Square Layer Normalization

Training

• Muon optimizer: Keller Jordan's Muon; Muon is Scalable for LLM Training
• Warmup-stable-decay schedule and 1 - sqrt cooldown: Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• PyTorch FlexAttention; Liger Kernel fused cross-entropy

Data and tokenizer

• Datasets: ClimbMix, DCLM-Edu, Nemotron-CC, NExtLong, CC-Math-Finest
• Tokenizer: Mistral-7B-v0.1 (Apache-2.0)

Evaluation

• lm-evaluation-harness; the Open SLM Leaderboard's Intelligence Index

---

License

Apache-2.0 for the weights, code and model card. The tokenizer is Mistral's, also Apache-2.0. Each training dataset keeps its own license.