kk-tokenizer-sp-bpe-32k

SentencePiece BPE tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000. Trained with sentencepiece on the cleaned Kazakh corpus Abzalbek89/corpus_clean (~1.5M documents).

SentencePiece BPE tokenizer for Kazakh (character_coverage=1.0, NFKC). Uses SentencePiece's whitespace-as-▁ representation.

Held-out fertility result

Metric	Value
Rank	2 / 13 (lower fertility is better)
Vocabulary size	32,000
Fertility (tokens / word)	1.758
Compression (chars / token)	4.463
Compression (bytes / token)	8.081
Total tokens / words	5,214,647 / 2,966,207

Evaluated on 2,966,207 whitespace-words from the validation split of Abzalbek89/corpus_clean (14,831+ documents, ~3M words). Full ranking and methodology: Abzalbek89/kk-tokenizer-fertility-baseline.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-sp-bpe-32k")
ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
                 add_special_tokens=False)
print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))

Companion artifacts

Training corpus: Abzalbek89/corpus_clean
Fertility benchmark + reproducible scripts: Abzalbek89/kk-tokenizer-fertility-baseline
Sibling tokenizers (BPE / Unigram / SP / morph-aware): see the dataset README for the full table.

Citation

@misc{kk_tokenizer_2026,
  title  = {Tokenizer Optimization for Kazakh Small Language Models},
  author = {Abzalbek Ulasbek},
  year   = {2026},
  howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-sp-bpe-32k}},
}

License

Apache 2.0