A 780M-parameter chat model trained from scratch as a personal learning project, packaged to run entirely in the browser with WebLLM 0.2.86 over WebGPU. This repository is a static file that the web demo downloads.
| Architecture | decoder-only transformer |
| Parameters | 778,782,720 (tied input/output embedding) |
| Layers / width | 40 layers, d_model 1280, SwiGLU 3584 |
| Attention | grouped-query, 20 query heads / 5 KV heads, head_dim 64, per-head QK-RMSNorm before RoPE (θ = 10,000) |
| Context | 4,096 tokens |
| Tokenizer | GPT-2 BPE (tiktoken's gpt2) + <|im_start|> 50257, <|im_end|> 50258. 50,259 tokens |
| Chat format | ChatML as trained, no BOS, no default system prompt, stops on <|im_end|> or <|endoftext|> |
Training: Pretrained on 30.0B tokens of FineWeb-Edu sample-100BT (final validation loss 2.5422 on a 1,048,576-token held-out subset), then supervised fine-tuning for 3 epochs on TuluTalk (Tülu 3 SFT and SmolTalk conversations).
Evaluation (of the unquantized checkpoint): (lm-eval 0.4.13, zero-shot, full sets, float32):
IFEval prompt-level strict 0.3771, instruction-level strict 0.4844. The
pretrained base model scores 0.4760 HellaSwag acc_norm.
q4f16_1: 4-bit group-quantized weights (group size 32, float16 scales) with float16 activations, compiled
for WebGPU with MLC-LLM.
model/params_shard_2.bin 33 MBmodel/params_shard_3.bin 32 MBmodel/params_shard_0.bin 32 MBmodel/params_shard_1.bin 32 MBmodel/params_shard_10.bin 30 MBmodel/params_shard_11.bin 30 MBmodel/params_shard_12.bin 30 MBmodel/params_shard_13.bin 30 MBmodel/params_shard_4.bin 30 MBmodel/params_shard_5.bin 30 MBmodel/params_shard_6.bin 30 MBmodel/params_shard_7.bin 30 MBmodel/params_shard_8.bin 30 MBmodel/params_shard_9.bin 30 MBmodel/params_shard_14.bin 8 MB