kk-tokenizer-unigram-32k

SentencePiece Unigram (v1) tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000. Trained with sentencepiece on the cleaned Kazakh corpus Abzalbek89/corpus_clean (~1.5M documents).

First-iteration SentencePiece Unigram tokenizer for Kazakh (character_coverage=0.9995). Superseded by kk-tokenizer-sp-unigram-32k. Kept for reproducibility of the v1 fertility experiment.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-unigram-32k")
ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
                 add_special_tokens=False)
print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))

Companion artifacts

• Training corpus: Abzalbek89/corpus_clean
• Fertility benchmark + reproducible scripts: Abzalbek89/kk-tokenizer-fertility-baseline
• Sibling tokenizers (BPE / Unigram / SP / morph-aware): see the dataset README for the full table.

Citation

@misc{kk_tokenizer_2026,
  title  = {Tokenizer Optimization for Kazakh Small Language Models},
  author = {Abzalbek Ulasbek},
  year   = {2026},
  howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-unigram-32k}},
}

License

Apache 2.0