Getok: Custom Tokenizer for Tibetan Buddhist Texts

This is a custom Byte Pair Encoding (BPE) tokenizer specifically for Tibetan Buddhist texts. It was trained using the SentencePiece / Hugging Face Tokenizers library. It is designed to tokenize text data efficiently for downstream NLP tasks. The tokenizer supports Unicode text in both Tibetan and English and was trained on a domain-specific corpus of Tibetan Buddhist texts.

This model was developed as part of the MLotsawa project. More information can be found here.

Getok was introduced in the paper Optimizing T5 for Lightweight Tibetan-English Translation (Moore & Lauren, 2025), where it is used as the tokenizer for the mlotsawa-ground-small and mlotsawa-ground-base translation models. The paper's code, including how this tokenizer was trained and evaluated, is available at optimizing-t5-tibetan-english-mt.

Special thanks to Andres Montano for suggesting the name of this tokenizer.

Use Cases

• Preprocessing for text classification, translation, summarization, or language modeling tasks
• Training/fine-tuning language models for Tibetan Buddhism related tasks.

Details

Tokenizer Type: BPE (Byte Pair Encoding)

Vocabulary Size: 32,000

Normalization: None

Special Tokens: "[PAD]", "[BOS]", "[EOS]", "<unk>"

Tokenization Level: Subword

Languages: Tibetan, English

Training Data

The tokenizer was trained on a corpus consisting of:

• 879,132 sentences of Tibetan from Buddhist texts
• 879,132 sentences of English from translations of the Tibetan texts

Usage

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained('billingsmoore/getok-v0')

tokenizer.encode('འཇམ་དཔལ་གཞོན་ནུར་གྱུར་པ་ལ་ཕྱག་འཚལ་ལོ༔')

Limitations

The tokenizer currently supports unicode text in Tibetan or Latin script. However, it was only trained on Tibetan and English texts and should not be expected to perform well on other languages that use those scripts (i.e. Dzongkha, French)

This tokenizer is not suitable for languages that are written in other scripts (i.e. Greek, Russian)

Finetuning a pretrained model using this tokenizer should be expected to take longer than finetuning using the model's own tokenizer because the model will need to adapt to the new encodings.

Citation

If you use this tokenizer, please cite the paper it was introduced in:

@article{moore2025optimizing,
  title   = {Optimizing T5 for Lightweight Tibetan-English Translation},
  author  = {Moore, Jacob and Lauren, Paula},
  year    = {2025},
  journal = {Research Square},
  doi     = {10.21203/rs.3.rs-7409829/v1},
  url     = {https://doi.org/10.21203/rs.3.rs-7409829/v1},
  note    = {Preprint}
}

Author & Contact

Author: billingsmoore

Contact: billingsmoore [at] gmail [dot] com