ZeroS-Pico-v1.1
This release improves upon ZeroS-Pico-v1.0 through architectural refinements and extended pretraining on over 1.7B tokens.
Architecture: What's different?
| Component | ZeroS-Pico-v1.0 | ZeroS-Pico-v1.1 | Notes |
|---|---|---|---|
| Layers | 9 | 12 | Deeper network depth |
| Hidden Size | 32 | 32 | No change |
| Vocab Size | 260 | 260 | No change |
| Attention Heads | 4 | 4 | No change |
| KV Heads | 2 | 2 | No change |
| Intermediate Size | 64 | 64 | No change |
| SwiGLU Interval | 4 | 6 | Same FFN capacity |
| mHC Lanes | 4 | 1 | Higher training throughput with virtually no loss in precision |
| Engram | True | False | Rerouted parameters into layers instead of Engram embeddings |
| Max Position Embeddings (training sequence length) | 96 | 155 | Increased maximum context |
| Tie Word Embeddings | True | True | No change |
ZeroS-Pico-v1.1 takes a deep-and-narrow approach, focusing parameter budget on raw transformer depth and throughput rather than auxiliary mechanisms like Engram embeddings and multi-lane routing.
Training Dataset
ZeroS-Pico-v1.1 was trained on 1.7B tokens (up from 600M in v1.0), covering web text, educational material, synthetic data, normalized code, and mathematics.
| Dataset | Share |
|---|---|
| FineWeb-Edu | 36.0% |
| DCLM Baseline 1.0 | 22.9% |
| FinePhrase | 13.4% |
| MGA FineWeb-Edu | 10.3% |
| Tiny Strange Textbooks | 8.2% |
| OpenMathInstruct-2 | 7.6% |
| NPset-2 Python-Edu | 1.6% |
Benchmark Results
| Task | ZeroS-Pico-v1.0 | ZeroS-Pico-v1.1 | Difference |
|---|---|---|---|
| ARC Challenge | 22.95% | 24.57% | +1.62% |
| ARC Easy | 27.65% | 26.30% | -1.35% |
| HellaSwag | 25.94% | 26.19% | +0.25% |
| PIQA | 49.62% | 51.14% | +1.52% |
| ArithMark-3.0 | 31.50% | 32.00% | +0.50% |
| Average | 31.53% | 32.04% | +0.51% |
As you can see, ZeroS-Pico-v1.1 is a steady improvement over v1.0, outperforming it on 4 out of 5 benchmarks with notable gains on ARC Challenge (+1.62%) and PIQA (+1.52%), despite a regression on ARC Easy.
On the Open SLM Leaderboard, at just 59.8K parameters, it outranks the 2.6M parameter Supra-Mini-v4-2M:
Hardware
- AMD Ryzen 5 2600
License
Apache 2.0
Citation
@misc{zeros-pico-v1.1,
title = {ZeroS-Pico-v1.1},
organization = {FromZero},
authors = {Paul Courneya},
year = {2026},
url = {https://huggingface.co/fromziro/ZeroS-Pico-v1.1}
}