ZeroS-v0.1-150M

This release marks v0.1, the initial beta release in the Zero model family. As an early experimental checkpoint, performance currently lags behind established small-scale baselines of comparable size (such as SmolLM2, MobileLLM, GPTX-2, and GPTX-2.5).

Architecture

Property	Value
Layers	34
Hidden size	576
Intermediate size	1555
Attention heads	18 (GQA kv=6, head_dim=32)
Full / Linear (DeltaNet) layers	34/0
DeltaNet conv kernel	4
DeltaNet kv heads	8/16
Partial rotary factor	0.25
Max sequence length	2048
Vocab size	32770
Tied embeddings	True
Total parameters	151.638M

Training

Tokens seen: 69,601,930,240
Val loss: 2.5434
Val PPL: 12.72

Benchmarks

Task	Metric	Score
HellaSwag	acc_norm	37.02%
PIQA	acc_norm	66.76%
ARC-Easy	acc_norm	48.70%
ARC-Challenge	acc_norm	28.84%
arithmark-3.0	acc_norm	35.80%

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("fromziro/ZeroS-v0.1-150M")
model = AutoModelForCausalLM.from_pretrained("fromziro/ZeroS-v0.1-150M")
inputs = tokenizer("Hello", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))