ZeroS-Micro-v1.0

ZeroS-Micro-v1.0 is a 3M-parameter SLM trained on 2.95B tokens of a high-quality dataset mix, featuring a custom architecture designed for parameter efficiency and representational quality at sub-10M scale.

Originally, this was supposed to be math-focused model called ZeroS-Micro-Math, but it started doing well on general language benchmarks during pretraining, so we've adopted it as our general micro model.

Architecture

The architecture of ZeroS-Micro-v1.0 is built around an interleaved feed-forward design that packs 18 layers into just 3 million parameters. Rather than placing dense feed-forward blocks at every layer, ZeroS-Micro-v1.0 alternates between parameter-free Hadamard FFNs and SwiGLU FFNs. This allows the model to achieve greater depth while bounding the total parameter count.

• Hidden Size: 144
• Vocab Size: 2564
• Number of Layers: 18 (9 SwiGLU, 9 Hadamard)
• Intermediate Size (SwiGLU FFN Capacity): 384
• Number of Attention Heads: 3
• Number of KV Heads: 1 (Grouped Query Attention / GQA)
• Dimensions Per Head: 48 (144 / 3)
• XSA Attention: true
• RoPE Theta: 7500.0
• Sequence Length: 1536 (can generate past this at inference)
• Tied Word Embeddings: true
• Total Parameters: 2,869,813 (2.87M)

Training Dataset

ZeroS-Micro-v1.0 was trained on 2.95 billion tokens consisting of high-density mathematical reasoning, formal logic, code, and academic knowledge.

Benchmark Results

License

Apache 2.0.

Citation

@misc{zeros-micro-v1.0,
  title        = {ZeroS-Micro-v1.0},
  organization = {FromZero},
  authors      = {Paul Courneya},
  year         = {2026},
  url          = {https://huggingface.co/fromziro/ZeroS-Micro-v1.0}
}