kk-tokenizer-hf-unigram-32k

Unigram tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000. Trained with huggingface/tokenizers on the cleaned Kazakh corpus Abzalbek89/corpus_clean (~1.5M documents).

Unigram tokenizer with byte-level pre-tokenization (matching kk-tokenizer-bpe-32k's pre-tokenizer for an apples-to-apples BPE-vs-Unigram comparison within the huggingface/tokenizers library).

Held-out fertility result

Metric	Value
Rank	6 / 13 (lower fertility is better)
Vocabulary size	32,000
Fertility (tokens / word)	2.026
Compression (chars / token)	3.872
Compression (bytes / token)	7.011
Total tokens / words	6,010,620 / 2,966,207

Evaluated on 2,966,207 whitespace-words from the validation split of Abzalbek89/corpus_clean (14,831+ documents, ~3M words). Full ranking and methodology: Abzalbek89/kk-tokenizer-fertility-baseline.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-hf-unigram-32k")
ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
                 add_special_tokens=False)
print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))

Companion artifacts

Training corpus: Abzalbek89/corpus_clean
Fertility benchmark + reproducible scripts: Abzalbek89/kk-tokenizer-fertility-baseline
Sibling tokenizers (BPE / Unigram / SP / morph-aware): see the dataset README for the full table.

Citation

@misc{kk_tokenizer_2026,
  title  = {Tokenizer Optimization for Kazakh Small Language Models},
  author = {Abzalbek Ulasbek},
  year   = {2026},
  howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-hf-unigram-32k}},
}

License

Apache 2.0