This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro-Dense. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.
- LS3-Micro-Dense-Base.bin: Weights 36.5 MB
- LS3-Micro-Dense-Base.bin.env: Training config: 355 B
- LS3-Micro-Dense-Base.bin.opt: Optimizer state: 36.4 MB
- LS3-Micro-Dense-Base.bin.opt.mu: Muon momentum: 36.4 MB
- LS3-Micro-Dense-Base.bin.opt.t1: Attention gate state 6.3 MB
LS3-Micro-Dense: First dense model in the LS family.
| (acc norm except LAMBADA) | HellaSwag | ARC-E | ARC-C | PIQA | LAMBADA | ArithMark | *II | **Intelligence Density |
|---|---|---|---|---|---|---|---|---|
| LS2.5-108M-A17M-Base | 28.24 | 36.66 | 23.21 | 57.40 | 17.89 | 37.00 | 9.89 | 0.231 |
| LS-Micro-Base (11.49M) | 27.41 | 34.72 | 21.93 | 54.84 | 11.06 | 35.50 | 7.24 | 0.909 |
| LS3-Micro-Base (9.97M) | 27.31 | 34.76 | 23.29 | 57.34 | 12.87 | 38.90 | 9.64 | 1.290 |
| LS3-Micro-Dense-Base (9.89M) | 28.53 | 39.27 | 20.31 | 56.58 | 16.15 | 47.50 | 11.99 | 1.212 |
With the help of a significantly bigger and curated dataset, LS3-Micro-Dense is able to achieve significant gains in LAMBADA, ARC-E and ArithMark. ARC-C remains below chance, while HellaSwag stays largely flat.
- Intelligence Index was calculated using the formula from Open SLM Leaderboard. (Results reproducable via Cudalm's built in bench tool)
- Intelligence density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))
- The model was benchmarked via Cudalm's LM eval tool.
| Model Specs | # |
|---|---|
| Shape | Dense transformer |
| Hidden Dim | 256 |
| Layers | 12 |
| Attention heads | 8 |
| KV heads | 2 |
| FFN | 660 |
| Context length | 2048 |
| Sliding window | 1024 |
| Vocab | 4096 (digits singular) |
| Optimiser | Muon 2D, AdamW else |
| Parameters | 9,890,048 (FP32) |
Training summary
| Stage A (stable) | Stage B (cooldown) | |
|---|---|---|
| Steps | 228,127 | 60,878 (total 289,005) |
| Tokens | 14.951B | 3.990B (total 18.94B) |
| Learning rate | 1.05e-3 held, 1,000-step warmup | 1.05e-3 falling to 2.1e-5 |
| Time, speed | 33 h, 128k tok/s | 8.3 h, 133k tok/s |
Datasets used:
Stage A (warmup and stable):
- HuggingFaceFW/fineweb-edu (8 shards, 7.2B tokens), ODC-By-1.0
- openbmb/Ultra-FineWeb, data/ultrafineweb_en (40 parts, 2.4B), ODC-By-1.0
- openbmb/Ultra-FineWeb-L3, qa (11 shards, 4B), Apache-2.0
- openbmb/Ultra-FineWeb-L3, multi_style (9 shards, 2.6B), Apache-2.0
- HuggingFaceTB/cosmopedia, data/web_samples_v2 (30 shards, 0.6B), Apache-2.0
- openbmb/Ultra-FineWeb-L3, multi_style (30 shards sifted from older corpus, 0.611B), ODC-By-1.0
- nvidia/OpenMathInstruct-2, refs/convert/parquet, default/train (8 shards, 1.5B), CC-BY-4.0
- HuggingFaceTB/finemath, finemath-4plus (40 shards, 1B), ODC-By-1.0
- ajibawa-2023/Education-College-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
- ajibawa-2023/Education-High-School-Students, refs/convert/parquet (2 shards, 0.3B), Apache-2.0
- ajibawa-2023/Education-Young-Children, refs/convert/parquet (2 shards, 0.2B), Apache-2.0
- HuggingFaceTB/cosmopedia: stanford, openstax, khanacademy (16 shards, 0.2B), Apache-2.0
- arithb: (self generated grade school level arithmetics, 0.9B), Self
Stage B (decay) on train splits:
- Rowan/hellaswag, default, train
- ARC-Easy, ARC-Challenge allenai/ai2_arc, ARC-Easy and ARC-Challenge, train
- github.com/ybisk/ybisk.github.io, master/piqa/data/train.jsonl and train-labels.lst
- allenai/openbookqa, additional, train
- allenai/sciq, default, train
- tau/commonsense_qa, default, train
- allenai/winogrande, winogrande_xl, train
Examples:
Temp 0.2
<|user|>
Do you like pineapple on pizza?
<|assistant|>
"Pineapple" is a pineapple-like fruit that is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is also known as "Pineapple" because it contains vitamin C and is rich in vitamin C.
Pineapple is a fruit that contains vitamin C and is rich in vitamin C. It is (truncated at 120 tokens)
Temp 0.7
<|user|>
How can I turn a chess piece into a stack of 100 dollar bills?
<|assistant|>
"Each dollar is divided into two parts. "
|Adding money to a stack of 100 dollars is $100+1 = 110$.
|Having a single stack of 100 dollars is $100+100 = 200$.
|Blending dollars is $110+100 = 210$.
|Glutening dollars is $100+100=200$.
|Bl (truncated at 120 tokens)
To try the model out yourself, benchmark it or fine tune it, clone Cudalm and build it with build.py (Blackwell 2.0 supported, or else build.py --cpu). Then download the LS3-Micro-Dense checkpoint in the files section and drop them into Cudalm's Models folder (folders will self init on first run).