LS1.5-63M-A16M
| task | 63M / 16M act | 92M / 22M act | 220M / 25M act | LS1.5 63M / 16M act | chance |
|---|---|---|---|---|---|
| arc_easy | 31.2% | 35.0% | 35.8% | 37.2% | 25.1% |
| hellaswag | 27.3% | 28.5% | 31.8% | 28.2% | 25.0% |
| piqa | 56.5% | 60.0% | 60.8% | 58.0% | 50.0% |
| lambada | 13.2% | 18.5% | 19.2% | 16.5% | 0% |
| mmlu | 24.8% | 23.8% | 24.5% | 26.5% | 25.0% |
Miniature mixture of experts model with top-1 routing. Trained entirely on a 1660 super.
Dataset is key (following the idea of Qwen3.8-27B):
LS1.5-63M-A16M has the exact same architecture as LS-63M-A16M. The only change made to the training process was training for longer on a bigger dataset (3B dataset compared to 1B, and for ~1.35 epoches). Overall, LS1.5-63M shows a modest improvement over LS-63M with minimal effort.
Total parameters: 63M Active parameters: 16M context length: 4096
Chat:
you> /reset
[context cleared]
you> hi
bot> Hello. How are you?
[10 tok, 333.3 tok/s, ctx 20/4096]
you> bad
bot> Thank you! I'm happy to assist you. How's the message you're seeing?
[23 tok, 621.6 tok/s, ctx 53/4096]
Knowledge:
you> explain dogs
bot> Dogs are dogs that have been shown to be a part of human-level relationships, and they are designed to be a powerful and efficient animal for the body. They typically go through the hunting and grooming of animals, and are often associated with human-food intolerances and the like. While dogs are undoubtedly important, they are also a significant part of human societies and are often associated with the natural world.
Dogs are mementos and playful, and they are often found in forest and hunting animals, and are often associated with the natural world. They often involve hunting, hunting, and even wildlife, and are often associated with the natural world. Dogs have been found in forests and harem, and are often associated with the natural world.
In summary, dogs are a great part of human-level relationships and are often associated with the natural world. They can be a powerful and efficient animal, and are often associated with the natural world. Dogs are a significant part of human-level relationships, and they are often associated with the natural world.
[246 tok, 510.4 tok/s, ctx 259/4096]
To try it yourself:
Download tinylm.exe here and LS1.5-63M-A16M-q8.bin (placed in \models), run command tinylm chat LS1.5-63M-A16M-q8 2048