Nawah-BERT-6M-v2

A 5,993,600-parameter Arabic BERT trained from scratch on 15B tokens. Same architecture as its 5B-token predecessor, v1 (hidden 128, 8 layers, 2 heads, ctx 2,048, 32K vocab) — only the token budget changed, and it changed everything. v1 is kept internal; every number it produced is reported below so the comparison is still legible.

v1 → v2: the token budget was the whole story

Held-out MLM on 1,024 chunks neither run trained on, standard 80/10/10:

	tokens	loss @15%	ppl	loss @30%	in-context top-1	top-5
v1	5B	6.0424	420.9	6.1053	2.7%	7.3%
v2	15B	3.3542	28.6	3.8419	37.2%	53.0%
corpus unigram	—	8.031	3,076	—	—	—

Perplexity improved 14.7× and single-mask recovery went from near-nothing to recovering the exact token 37% of the time.

A correction worth recording

The v1 model card said it had saturated — that "5B tokens is roughly 500× past Chinchilla-optimal for this size, and the flat tail is what that looks like." That was wrong. The flat tail was the cosine learning rate decaying to zero, not the model running out of capacity. Warm-starting at lr 3e-4 on fresh tokens moved held-out loss 6.04 → 3.35. A 1.9M-parameter backbone had far more to give than its own loss curve suggested — a flat loss late in a cosine schedule says almost nothing about capacity.

Training

Continued from v1: 10B unseen tokens (permutation chunks 2,441,344 onward of tokens_20B_sep.bin, so no repetition of what v1 saw), lr 3e-4 cosine, 38,146 steps at 262,144 tokens/step, mask 30%→15% annealed at step 33,949, bf16, torch_compile. 3.5 h on a single consumer GPU. Total across both runs: 15B of the corpus's 20B.

Downstream

Fine-tuned on three tasks and compared against the Llama models built for each. Full numbers in each card:

task	v1	v2	reference (Llama)
RuleCheck unseen wording	0.9357	0.9598	0.9949 (5M)
Guard held-out over-refusal ↓	0.0146	0.0115	0.0094 (52M)
Router unseen category sets	0.9137	0.9327	0.9308 (52M)

Use mean pooling, not [CLS]

Pretrained on packed 2,048-token chunks with no [CLS], so position 0 has no summary role and BertForSequenceClassification's pooler arrives randomly initialised. With it the model never leaves the class prior at any learning rate from 1e-4 to 3e-3. Mean-pool over non-pad positions instead — see bert_meanpool.py in the task repos.

oddadmix/Nawah-BERT-6M-v2