nanochat-40M


Params	39.9M
Arch	decoder-only transformer — 10 layers, d_model 512, GQA (8Q/4KV heads, head_dim 64), SwiGLU FFN 1536, RMSNorm (eps 1e-5), RoPE (θ=10000), tied embeddings
Vocab	16,384 byte-level BPE, ChatML (`<\|im_start\|>`/`<\|im_end\|>`)
Context	1,024
Precision	bf16

Files

File	Size	Notes
`model.safetensors`	76MB	HF-format weights (Llama-compatible layout)
`tokenizer.json`	–	16k ChatML BPE
`nanochat-40m-bf16.gguf`	77MB	full-precision GGUF
`nanochat-40m-q8_0.gguf`	41MB	Q8_0
`nanochat-40m-q4_k_m.gguf`	25MB	Q4_K_M
`sft_smoltalk_40m_laptop.yaml`	–	full training config

Training data

HuggingFaceTB/smol-smoltalk, subset by source, 1 epoch, ≤1,024 tokens/sample:

source	conversations	samples	tokens
everyday-conversations	2,252	8,595	933,426
explore-instruct-rewrite	3,020	5,986	271,456
smol-summarize-5k	4,658	18,514	6,666,822
smol-summarize-20k	18,839	33,893	5,079,395
smollm-rewrite-30k	27,343	39,810	9,410,604
smol-contraints	34,107	57,443	7,280,378
openhermes-50k	45,130	67,336	16,686,328
smol-magpie-ultra-short	25,934	25,934	33,671,827
total	161,283	257,511	80,000,236

Train: 206,644 samples / val: 512 (deterministic shuffle, seed 0).

Training

Trained on an Apple M3 Max (30-core GPU) in ~3.1 hours (~8.3k tok/s) on 80.0M tokens from HuggingFaceTB/smol-smoltalk. Could have gone a lot faster, but I was using my GPU for a lot of other things all at once. AdamW (lr 2e-3 cosine, warmup 200, wd 0.1, clip 1.0), effective batch 32, 6,458 steps. Val loss 2.66.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("hudsongouge/nanochat-40M")
m = AutoModelForCausalLM.from_pretrained("hudsongouge/nanochat-40M", dtype=torch.bfloat16)
prompt = "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
ids = tok(prompt, return_tensors="pt").input_ids
print(tok.decode(m.generate(ids, max_new_tokens=40)[0][ids.shape[1]:]))

Or point llama.cpp / LM Studio / Ollama at the GGUFs.

Examples (greedy)

Q: What is the capital of France?
A: The capital of France is Paris.

Q: Explain photosynthesis in one sentence.
A: Photosynthesis is a process used by plants to convert light energy into chemical energy.

Q: Write a haiku about the ocean.
A: In the vast ocean's shade,
the waves dance under moonlight,
silent echoes fade.

Q: What is 17 + 25?
A: To find the sum of 17 and 25, we can add them together:
17 + 25 = 17 + 25 = 30.

Dumb, but decent.

加载中…

Params

39.9M

Arch

decoder-only transformer — 10 layers, d_model 512, GQA (8Q/4KV heads, head_dim 64), SwiGLU FFN 1536, RMSNorm (eps 1e-5), RoPE (θ=10000), tied embeddings

Vocab

16,384 byte-level BPE, ChatML (<|im_start|>/<|im_end|>)

Context

1,024

Precision

bf16

Files

File	Size	Notes
`model.safetensors`	76MB	HF-format weights (Llama-compatible layout)
`tokenizer.json`	–	16k ChatML BPE
`nanochat-40m-bf16.gguf`	77MB	full-precision GGUF
`nanochat-40m-q8_0.gguf`	41MB	Q8_0
`nanochat-40m-q4_k_m.gguf`	25MB	Q4_K_M
`sft_smoltalk_40m_laptop.yaml`	–	full training config

Training data

HuggingFaceTB/smol-smoltalk, subset by source, 1 epoch, ≤1,024 tokens/sample:

source	conversations	samples	tokens
everyday-conversations	2,252	8,595	933,426
explore-instruct-rewrite	3,020	5,986	271,456
smol-summarize-5k	4,658	18,514	6,666,822
smol-summarize-20k	18,839	33,893	5,079,395
smollm-rewrite-30k	27,343	39,810	9,410,604
smol-contraints	34,107	57,443	7,280,378
openhermes-50k	45,130	67,336	16,686,328
smol-magpie-ultra-short	25,934	25,934	33,671,827
total	161,283	257,511	80,000,236

Train: 206,644 samples / val: 512 (deterministic shuffle, seed 0).

Training

Use

from transformers import AutoModelForCausalLM, AutoTokenizer import torch tok = AutoTokenizer.from_pretrained("hudsongouge/nanochat-40M") m = AutoModelForCausalLM.from_pretrained("hudsongouge/nanochat-40M", dtype=torch.bfloat16) prompt = "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" ids = tok(prompt, return_tensors="pt").input_ids print(tok.decode(m.generate(ids, max_new_tokens=40)[0][ids.shape[1]:]))

Or point llama.cpp / LM Studio / Ollama at the GGUFs.

Examples (greedy)

Q: What is the capital of France? A: The capital of France is Paris. Q: Explain photosynthesis in one sentence. A: Photosynthesis is a process used by plants to convert light energy into chemical energy. Q: Write a haiku about the ocean. A: In the vast ocean's shade, the waves dance under moonlight, silent echoes fade. Q: What is 17 + 25? A: To find the sum of 17 and 25, we can add them together: 17 + 25 = 17 + 25 = 30.

Dumb, but decent.