Benchmarks:
piqa:
acc,none: 0.6273
acc_norm,none: 0.6224
hellaswag:
acc,none: 0.2922
acc_norm,none: 0.3220
arc_easy:
acc,none: 0.4987
acc_norm,none: 0.4491
arc_challenge:
acc,none: 0.2099
acc_norm,none: 0.2500
arithmark_3.0:
acc,none: 0.3390
acc_norm,none: 0.3390
Int Index based on OpenSLM-Leaderboard: 15.09
It matches or slightly beats GPT2-Small on most Benchmarks
Architecture:
Context Lenght: 1024
Factorized & Tied Embeddings
Vocabulary Size: 32768
Embedding Rank: 384
GQA Attention (2:1 Head ratio)
10 Query and 5 Key/Value heads
Head Dimension: 64
Positional Encoding: RoPE (θ=10000)
QK-Norm
30 identical Transformer Layers
Hidden Size: 640
Intermediate Size: 1728
Activation: SwiGLU
Training:
Muon and AdamW
Around 150M Parameters
7 Billion tokens of Pretraining
11 Hours on an RTX Pro 6000
Quick Start
You can use the Inference.py script.
Example command: inference.py --Prompt "The capital of Denmark is" --temperature 0.1 --max_new_tokens 32
Output: The capital of Denmark is Copenhagen, which has a population of about 1.5 million people. The city is located in the northern part of the country and is surrounded by the sea.