LS-63M-A16M
| task | 63M / 16M act | 92M / 22M act | 220M / 25M act | chance |
|---|---|---|---|---|
| arc_easy | 31.2% | 35.0% | 35.8% | 25.1% |
| hellaswag | 27.3% | 28.5% | 31.8% | 25.0% |
| piqa | 56.5% | 60.0% | 60.8% | 50.0% |
| lambada | 13.2% | 18.5% | 19.2% | 0% |
| mmlu | 24.8% | 23.8% | 24.5% | 25.0% |
Miniature mixture of experts model with top-1 routing. Trained entirely on a 1660 super. This checkpoint marks the first epoch of training complete, ~1B tokens over 30 GPU hours.
Total parameters: 63M Active parameters: 16M context length: 4096
Training end evals:
| val loss | 1.4286 |
|---|---|
| perplexity | 4.17 |
Chat:
you> hi
bot> Hello! How can I assist you today?
[13 tok, 117.1 tok/s, ctx 23/16384]
you> what is the capital of france?
bot> Juan Van Gogh
[12 tok, 129.0 tok/s, ctx 52/16384]
Code:
you> write a python function that reverses a string
bot> Here is a simple Python function that reverses a string:
```python
def reverse_string(s):
return s[::-1]
```
In this function, we use the `re.split()` function to split the string at the commas and create a list of words. Then we use `re.split()` to split the string on the `^`, and finally, we use `str.split()` to split the list of words.
[98 tok, 63.9 tok/s, ctx 116/16384]
To try it yourself:
Download tinylm.exe and LS-63M-A16M-q8.bin (placed in \models), run command tinylm chat LS-63M-A16M-q8 2048
Note: the bundled tinylm.exe is likely outdated. For the latest version, check here for the source code of tinylm and windows prebuilts.