KW5-230M

The third-generation KW5 base model: a Kiswahili language model pretrained from scratch, tokenizer included, on 7.90B tokens of Kiswahili. Built by Regnant.

A base model continues text. It does not follow instructions; for that, use KW5-230M Instruct, which was fine-tuned from this checkpoint. This one is for research, for scoring Kiswahili text, and as the starting point for your own fine-tune.

230,423,424 parameters, the input embedding and the output head tied.

How it compares

Scored in float32 on one T4 (2026-10-28, seed 20260914). Bits per byte is the quantity that compares models with different tokenizers; perplexity does not.

How to read it:

• Bits per byte is measured on 120 MasakhaNEWS test articles (320,701 bytes). News is in KW5's pretraining mix and overlap was not checked, so the KW5 rows are in-domain. A Kiswahili-only GPT-2 trained on 100 MB comes within 0.003 of this model. Compression is not where a 230M model separates itself.
• Belebele (900 items, reading, four choices): every model beats chance, and this one's interval just clears the second generation's.
• AfriXNLI (600, inference, three choices): no model beats chance. This one comes closest, at p = 0.055.
• MasakhaNEWS (476, news topic, seven choices) is in-domain for the same reason as bits per byte.

On its own held-out validation split (545 documents across every register), bits per byte fell from 0.859 at the end of the stable phase to 0.813 after the decay. By register: arithmetic 0.07, chat 0.28, encyclopedia 0.79, news 0.79, PDFs 0.80, parliament 0.85, translated educational 0.88, religious 0.95, web 0.96, mixed long-form 1.43. The split is in-distribution for this model only, so these numbers do not compare it with anything else.

Quick start

!pip install -q huggingface_hub sentencepiece safetensors

import sys, torch, sentencepiece as spm
from huggingface_hub import snapshot_download

path = snapshot_download("regnant-io/kw5-230M")             # ~1.04 GB, cached
sys.path.insert(0, path)          # the repo ships modeling_kw5v2.py and kw5_tokenizer.py
from modeling_kw5v2 import KW5V2ForCausalLM
from kw5_tokenizer import make_encoder

model = KW5V2ForCausalLM.from_pretrained(path)
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
encode = make_encoder(sp)

torch.manual_seed(0)
ids = encode("Tanzania ni nchi ya Afrika Mashariki.")       # no <s>: pretraining had none
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_p=0.9,
                     repetition_penalty=1.1, eos_id=2)
print(sp.decode(out))

Output on CPU, as run against these weights:

Tanzania ni nchi ya Afrika Mashariki. Nchi hii inapakana na Kenya, Uganda,
Rwanda na Burundi. Ni nchi yenye historia nzuri sana ya karne nyingi.

 Historia

 Historia ya awali
 Kulikuwa na falme nyingi tofauti katika Afrika Mashariki wakati wa ukoloni.
Mojawapo ya falme hizo ilikuwa Ufalme wa Buganda ambao ulitawala eneo la
Uganda kuanzia

Fluent, shaped like an encyclopedia article (a heading, a subheading), and wrong in the details: Tanzania also borders Mozambique, Malawi, Zambia, the DRC and the ocean, and the kingdoms came long before colonial rule. sp.decode leaves a space at the start of each line the encoder split; strip it if it matters.

Encode with kw5_tokenizer.py, not sp.encode. It encodes line by line, so the first word of every line keeps its word-boundary piece, which is how the corpus was packed. sp.encode on a multi-line string splits the first word after each newline into pieces the model rarely saw. Do not prepend <s>: documents were packed without it.

A stock AutoModelForCausalLM will not load it: the block has Canon layers. modeling_kw5v2.py needs only PyTorch and safetensors.

What it does

Greedy, in the pretraining layout, before any fine-tuning:

• Arithmetic with its working. 56 of 57 bare sums, 56 of 60 expressions with brackets (slips in long multiplication) and 44 of 60 word problems, written out column by column as the generated arithmetic register taught it. The decay, which doubled that register, bought 7 and 5 of those items.
• Fluent Kiswahili on news, encyclopedia and educational prose.
• The chat format, unprompted. Its decay mixed chat four times over, so it sometimes writes <|user|> and <|end|> blocks of its own. That is what made the instruct fine-tune cheap; in a plain continuation, stop on <|end|> (id 7) as well as </s>.

What it gets wrong

• Facts. It states Kilimanjaro's height and Nyerere's birthday wrongly and confidently.
• Loops on open prompts: four of ten greedy samples repeated themselves (3-gram repetition 0.64 to 0.87). Sample with a repetition penalty.
• Scope. Kiswahili only, mostly Tanzanian, partly machine-translated corpus. No safety tuning.

Training

The size was not chosen by taste. A throughput sprint measured three shapes on the TPU, and the rule was the largest one that reaches 30 training tokens per stored parameter in the hours available. The 230M shape ran at 57,109 tokens per second in the first session's calibration, and the token budget was locked there and stored in every checkpoint.

Before a single step ran, a checkpoint past step 200 had to reload in a fresh process and reproduce its loss, and greedy samples were read every 500 steps. The second generation lost about 27 TPU-hours to a reload path nobody had tested; this one tested it first.

Citation

@misc{regnant2026kw5v3,
  title  = {KW5-230M: a Kiswahili language model from the tokenizer up},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-230M}
}

Apache 2.0.