Somali ByteLevel BPE v2

A 48,000-token byte-level BPE tokenizer for Somali, trained by SOMAST for the Somali NLP Corpus.

• Byte-level, so every input round-trips exactly and there is no unknown token.
• 48,000 types, including special tokens <|endoftext|>, <|pad|>, <|im_start|>, <|im_end|> and 12 reserved slots for a decoder-LM control set.
• Context length set to 8,192 in tokenizer_config.json.

Use

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("somast/somali-bpe-v2")
ids = tok.encode("Caafimaadka carruurtu waa mudnaanteenna koowaad.")
print(tok.convert_ids_to_tokens(ids))
# ['Caafimaadka', 'Ġcarruurtu', 'Ġwaa', 'Ġmudnaan', 'teenna', 'Ġkoowaad', '.']
print(tok.decode(ids))  # the input, exactly

Results

Scored on a held-out split of 63,542 documents (about 0.8% of the Somali NLP Corpus 1.0.0, excluded by a deterministic hash of the document id). Mean tokens per word, lower is better:

Over the whole 1.0.0 corpus (7,913,008 documents, 826,973,689 words) this tokenizer produces 1,129,084,663 tokens, 1.3653 per word. Per-source figures and the full method are in the repository.

Limits

• The vocabulary was trained on an earlier eleven-source build of the corpus (2026-09-01) and was not retrained for the six sources added afterwards (Glot500, Somali Web Corpus, FinePDFs, FineWeb-2, Somali Alpaca, Somali TinyStories). FinePDFs tokenizes above the holdout mean (1.4905), so a retrained vocabulary should help on PDF-style text.
• Tuned for Somali. Other languages work, because the tokenizer is byte-level, but are split inefficiently.
• Only the tokenizer is licensed here. The corpus it was trained on is released separately, under the per-record licenses described on its card.

Citation

@misc{somali_bpe_v2,
  title  = {Somali ByteLevel BPE v2},
  author = {SOMAST},
  year   = {2026},
  url    = {https://huggingface.co/somast/somali-bpe-v2}
}

Contact: info@somast.so