Loadingβ¦
Loadingβ¦
myX-Tokenizer is a high-efficiency tokenizer specifically engineered for the Burmese (Myanmar) language, with extended support for English and Pali. It is designed to overcome the "fertility" issues common in multilingual tokenizers (like mBERT and XLM-R), where Burmese text is often over-fragmented into nearly character-level subwords.
Developed by Khant Sint Heinn (Kalix Louis) and released under DatarrX, this tokenizer provides a balanced representation of the Burmese language, optimized for modern Large Language Models (LLMs) and Embedding models.
nmt_nfkc_cf)<unk> tokens)The tokenizer was trained on a massive 40M+ line corpus with a strategic weight distribution to ensure cross-domain robustness:
| Language/Domain | Percentage | Sources |
|---|---|---|
| Burmese (General) | ~70% | Wikipedia, News (BBC Burmese), Literature, Written/Spoken Text |
| English | ~20% | High-quality English general sentences |
| Pali | ~10% | Tipitaka Dataset (Religious and Historical texts) |
While the model is highly capable in Unicode formal and informal Burmese, users should note:
The primary goal of myX-Tokenizer is to optimize the "Token-to-Character" ratio for Myanmar NLP. By reducing the Fertility Rate, the model can process significantly more semantic information within the same context window compared to general multilingual tokenizers.
To ensure an unbiased evaluation, we benchmarked the tokenizers using the jojo-ai-mst/Myanmar-Agricutlure-1K dataset (specifically the "Output" column with 1,053 rows). Note: This dataset was held out during the training phase of myX-Tokenizer to serve as a clean test set.
| Tokenizer | Avg Tokens per Sentence | Efficiency Score (Fertility) |
|---|---|---|
| myX-Tokenizer | 34.2 | Most Efficient |
| XLM-R | 64.05 | - |
| mBERT | 109.59 | - |
The following example demonstrates how each model segments a complex Burmese sentence. While multilingual models struggle with Burmese character boundaries, myX-Tokenizer preserves morphological and semantic integrity.
Test Sentence:
"αα°αα»αα―αΈαα αΊαα»αα―αΈα α α¬αα±αααΊαα»α±αΈααΎα― ααααΊαα±α¬αα»α±α¬ααΊαα½ααΊααΌααΊαΈαααΊ ααα―αα°αα»αα―αΈ αααΉαα¬α·ααΌα±αα―αΆαα±α«αΊααΎ αα»α±α¬ααΊαα½ααΊαα½α¬αΈααΌααΊαΈαααΊ ααΌα αΊαααΊα"
myX-Tokenizer (17 tokens):
['βαα°αα»αα―αΈ', 'αα
αΊαα»αα―αΈ', 'α', 'βα
α¬αα±', 'αααΊαα»α±αΈααΎα―', 'βααααΊαα±α¬', 'αα»α±α¬ααΊαα½ααΊ', 'ααΌααΊαΈαααΊ', 'βααα―', 'αα°αα»αα―αΈ', 'βαααΉαα¬α·', 'ααΌα±αα―αΆ', 'αα±α«αΊααΎ', 'βαα»α±α¬ααΊαα½ααΊ', 'αα½α¬αΈααΌααΊαΈ', 'αααΊ', 'βααΌα
αΊαααΊα']
(Observation: Highly semantic and readable units.)
XLM-R (36 tokens):
['β', 'αα°αα»αα―αΈ', 'αα
αΊ', 'αα»αα―αΈ', 'α', 'βα
α¬', 'αα±', 'α', 'α', 'αΊ', 'αα»', 'α±αΈ', 'ααΎα―', 'β', 'αα', 'ααΊ', 'αα±α¬', 'αα»α±α¬ααΊ', 'αα½ααΊ', 'ααΌααΊαΈ', 'αααΊ', 'βααα―', 'αα°αα»αα―αΈ', 'βαααΉαα¬α·', 'ααΌα±', 'αα―αΆ', 'αα±α«αΊ', 'ααΎ', 'β', 'αα»α±α¬ααΊ', 'αα½ααΊ', 'αα½α¬αΈ', 'ααΌααΊαΈ', 'αααΊ', 'β', 'ααΌα
αΊαααΊα']
(Observation: Excessive fragmentation of simple words like 'αααΊαα»α±αΈααΎα―'.)
mBERT (68 tokens):
['α', '##α°', '##α', '##α»αα―αΈ', '##α', '##α
αΊ', '##α', '##α»αα―αΈ', 'α', 'α
', '##α¬', '##α', '##α±', '##α', '##α', '##αΊ', '##α', '##α»', '##α±αΈ', '##ααΎα―', 'α', '##α', '##α', '##αΊ', '##α', '##α±α¬', '##α', '##α»', '##α±α¬ααΊ', '##α', '##α½', '##ααΊ', '##ααΌααΊαΈ', '##αααΊ', 'α', '##αα―', '##α', '##α°', '##α', '##α»αα―αΈ', 'α', '##α', '##αΉ', '##α', '##α¬', '##α·', '##α', '##αΌ', '##α±', '##α', '##α―αΆ', '##α', '##α±α«αΊ', '##ααΎ', 'α', '##α»', '##α±α¬ααΊ', '##α', '##α½', '##ααΊ', '##α', '##α½', '##α¬αΈ', '##ααΌααΊαΈ', '##α', '##ααΊ', 'ααΌα
αΊαααΊ', 'α']
(Observation: Severe over-fragmentation; almost character-level, leading to high computational cost and loss of context.)
For the best results in your Myanmar NLP projects, we recommend using the tokenizer.model file directly with the sentencepiece library.
import sentencepiece as spm
# Load the model
sp = spm.SentencePieceProcessor(model_file='tokenizer.model')
# Encode
text = "ααΌααΊαα¬ααα―ααΊααΆα ααααΊαΈαα°αΈαα»α¬αΈ ααΎαα·αΊ Artificial Intelligence αααΊαΈααα¬"
tokens = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)
print(f"Tokens: {tokens}")
print(f"IDs: {ids}")
The tokenizer includes pre-defined IDs for modern AI workflows:
<|im_start|>, <|im_end|>, <|system|>, <|user|>, <|assistant|><|thought|>, <|reflection|><|context_start|>, <|tool_call|>, <|tool_response|>| Metric | Ultra Result |
|---|---|
| Total Characters | 40,000,000+ |
| Density (Chars/Token) | ~2.5 - 3.1 |
| Vocab Coverage (%) | 100% (Byte Fallback active) |
If you utilize this tokenizer in your research or applications, please use the following citation:
@software{khantsintheinn2026myxtokenizer,
author = {Khant Sint Heinn},
title = {myX-Tokenizer: A Specialized Unigram Tokenizer for Myanmar NLP},
year = {2026},
publisher = {DatarrX},
url = {https://huggingface.co/DatarrX/myX-Tokenize)}
}
Khant Sint Heinn, working under the name Kalix Louis, is a Machine Learning Engineer focused on Natural Language Processing (NLP), data foundations, and open-source AI development. His work is centered on improving support for the Burmese (Myanmar) language in modern AI systems by building high-quality datasets, practical tools, and scalable infrastructure for language technology.
He is currently the Lead Developer at DatarrX, where he develops data pipelines, manages large-scale data collection workflows, and helps create open-source resources for researchers, developers, and organizations. His experience includes data engineering, web scripting, dataset curation, and building systems that support real-world machine learning applications.
Khant Sint Heinn is especially interested in advancing low-resource languages and making AI more accessible to underrepresented communities. Through his open-source contributions, he works to strengthen the Burmese (Myanmar) tech ecosystem and provide reliable building blocks for future language models, search systems, and intelligent applications.
His goal is simple: to turn limited language resources into practical opportunities through clean data, useful tools, and community-driven innovation.
Connect with the Author:
GitHub | Hugging Face | Kaggle
Disclaimer: This project is an ongoing effort to improve Myanmar language support in AI. We welcome feedback and contributions via the Community tab.