LS2.5: Efficientmaxxing at its finest
LS2.5 Expands upon LS2 by introducing a new 16k vocab tokenizer instead of the 8K vocab. This increases efficiency (same dataset is ~12% less tokens than 8k variant) but active parameters also increases (~3M increase).
A new 5.7B pretrain dataset was created with the following sources:
- Ultrafineweb
- Github-python
- Cosmopedia-v2
- ufw-l3
- finemath
- hermes-2.5
- ultrachat
- toolace
- hermes-fc
- glaive-v2
- heremes-single
- heremes-gl5k
- apigen-mt
LS2.5-108M-A17M-Base
This repo contains the Base (pre-sft) checkpoint of LS2.5-108M-A17M.
A 108M MoE model with top-2 experts routing, entirely trained on ~5.7B tokens with a 5060ti over ~26h.
| task (acc_norm) | LS-220M-A25M | LS2-266M-A14M | LS2.5-108M-A17M-Base | LS2.5-108M-A17M-Coder | LS2.5-108M-A17M-Chat | chance |
|---|---|---|---|---|---|---|
| arc_easy | 35.8% | 36.2% | 35.8% | 34.4% | 33.8% | 25.1% |
| hellaswag | 31.8% | 30.0% | 29.2% | 26.6% | 27.4% | 25.0% |
| piqa | 60.8% | 60.0% | 57.4% | 58.4% | 58.0% | 50.0% |
| lambada | 19.2% | 19.6% | 14.4% | 16.6% | 18.6% | 0% |
LS2.5-108M-A17M retains 94% of benchmark accuracy of both models flagship models while being 2.4 times smaller.
Internal benchmark:
118 Multiple choice questions and 24 open prompts was benched on LS2.5-108M-A17M-Base and the two sft variants. The multiple choice questions covered facts, common sense, following instructions, picking-on-the-topic reply, staying consistent with the conversation and simple arithmetic. Random guessing scores 25%.
The model responds to the open prompts with 4 samples + 1 greedy reply. They are checked for the following:
- ends by itself, or hit the 128 token cap
- if it mentions the topic
- if it loops
- how varied wording is
- how the reply depends on the prompt vs other topics
| model | Multiple choice (length-adjusted) | Ends on its own | Loops | Wording variety | Depends on the question |
|---|---|---|---|---|---|
| LS2.5-108M-A17M-Chat | 59.3% | 49% | 1% | 0.93 | +0.47 |
| LS2.5-108M-A17M-Coder | 55.9% | 34% | 5% | 0.90 | +0.40 |
| LS2.5-108M-A17M-Base | 55.9% | 42% | 1% | 0.89 | +0.38 |
Overall, the fine tuned models don't gain any knowledge (benchmarks pretty much unchanged) but rather their output format and reply length.
| Model specs | |
|---|---|
| Total parameters: | 108M |
| Model parameters: | 66M |
| Active parameters: | 17M |
| Ngram table: | 42M |
| Context length: | 1024 |
| Pretrain details | |
| Tokens seen: | ~5.7B |
| GPU hours (5060ti): | ~26h |
Examples:
"chat" (temp 0.2)
you> hi
bot> You're welcome! If you have any other questions, feel free to ask.
[16 tok, 1454.5 tok/s, ctx 26/4096]
you> What is the capital of France?
bot> The capital of France is Pope Francis.
[10 tok, 1250.0 tok/s, ctx 26/4096]
you> which colour is the sky?\nA:blue\nB:green\nC:purple\nD:pink
bot> The sky is a beautiful and vibrant colors that make it look like a blue. The sky has a bright, warm glow, while the green light creates an airy atmosphere. It's also a natural beauty place to enjoy in your life.
[48 tok, 1090.9 tok/s, ctx 85/4096]
Usage:
To run this model, download the latest tinylm from github and build from source. Currently working on windows prebuilts.
Then run tinylm chat LS2.5-108M-A17M-Base-Q4_K_M.gguf 512 temp. Try temp 0.2 for general chat and 0.7 for nonsensical yap.