albert-khmer-codeswitch-tokenizer

This model is a Khmer-English Code-Switching ALBERT based on seanghay/albert-khmer-small, further pre-trained on a massive bilingual corpus with explicit code-switching data.

The model excels at understanding and processing mixed Khmer-English text, making it ideal for applications like sentiment analysis, named entity recognition, and question answering in code-switched contexts.

Model Details

• Developed by: NOSON
• Model type: ALBERT (A Lite BERT)
• Language(s): Khmer (km), English (en)
• License: Apache-2.0
• Finetuned from model: seanghay/albert-khmer-small

Model Description

This model extends the original Khmer ALBERT with enhanced capabilities for:

• Code-switching: Handles mixed Khmer-English text.
• Technical vocabulary: Better understanding of tech terms, URLs, and modern language.
• Cultural preservation: Maintains high integrity of complex Khmer words.

Training Data

The model was trained on a carefully balanced corpus consisting of:

• Khmer Wikipedia (2023-2024 dumps)
• WikiText-103 (English)
• MC4 (Khmer & English subsets)
• Custom Code-Switching Corpus (injected synthetic examples)

Training Procedure

Tokenizer Specifications

• Algorithm: SentencePiece Unigram
• Vocabulary Size: 64,000 tokens
• Character Coverage: 1.0 (ensures perfect handling of all Khmer diacritics and subscripts)
• Special Handling: Forced vocabulary injection for critical technical terms.

Evaluation Results

Lower TPC is better; higher percentages are better for integrity.

Complex Word Integrity Breakdown

🔍 Tokenization Examples

The true power of this custom 64K tokenizer is visible in how it handles complex Khmer morphology, mixed scripts, and English words without unnecessary fragmentation.

Click to expand detailed tokenization comparisons

1. Mixed Script Words

Text: The Khmer-English dictionary is very ល្អណាស់។

• ✅ OUR (11 tokens): ['The', 'Khmer', '-', 'English', '', 'dictionary', 'is', 'very', '', 'ល្អណាស់', '។']
• ❌ ALBERT (18 tokens): ['The', 'Kh', 'mer', '-', 'E', 'ng', 'li', 'sh', '', 'dict', 'ion', 'ary', 'is', 'very', '', 'ល្អ', 'ណាស់', '។'] (Fractures "Khmer", "English", "dictionary")
• ⚠️ XLM-R (12 tokens): ['The', 'Khmer', '-', 'English', 'di', 'ction', 'ary', 'is', 'very', 'ល្អ', 'ណាស់', '។']

2. Pure Khmer (Complex Compound)

Text: ព្រះពុទ្ធសាសនាមានសារៈសំខាន់។

• ✅ OUR (3 tokens): ['ព្រះពុទ្ធសាសនា', 'មានសារៈសំខាន់', '។'] (Perfectly intact)
• ❌ ALBERT (4 tokens): ['', 'ព្រះពុទ្ធសាសនា', 'មានសារៈសំខាន់', '។']
• ❌ XLM-R (8 tokens): ['ព្រះ', 'ពុទ្ធ', 'សាសនា', 'មាន', 'សារ', 'ៈ', 'សំខាន់', '។'] (Severely fractured)

3. Pure Khmer (Formal)

Text: នាយករដ្ឋមន្ត្រីបានថ្លែងសុន្ទរកថា

• ✅ OUR (3 tokens): ['នាយករដ្ឋមន្ត្រី', 'បានថ្លែង', 'សុន្ទរកថា']
• ✅ ALBERT (3 tokens): ['នាយករដ្ឋមន្ត្រី', 'បាន', 'ថ្លែងសុន្ទរកថា']
• ❌ XLM-R (9 tokens): ['', 'នាយករដ្ឋមន្ត្រី', 'បាន', 'ថ្', 'លែង', 'សុ', 'ន្ទ', 'រក', 'ថា'] (Fractures sub-consonants)

4. Pure Khmer (Long Sentence)

Text: ព្រះពុទ្ធសាសនាគឺជាសាសនាចម្បងក្នុងប្រទេសកម្ពុជា។

• ✅ OUR (6 tokens): ['ព្រះពុទ្ធសាសនា', 'គឺជា', 'សាសនា', 'ចម្បង', 'ក្នុង', 'ប្រទេសកម្ពុជា។']
• ❌ ALBERT (8 tokens): ['', 'ព្រះពុទ្ធសាសនា', 'គឺជា', 'សាសនា', 'ចម្បង', 'ក្នុង', 'ប្រទេសកម្ពុជា', '។']
• ❌ XLM-R (10 tokens): ['ព្រះ', 'ពុទ្ធ', 'សាសនា', 'គឺជា', 'សាសនា', 'ចម្', 'បង', 'ក្នុង', 'ប្រទេសកម្ពុជា', '។']

5. Pure English (Complex)

Text: The quick brown fox jumps over the lazy dog.

• ✅ OUR (10 tokens): ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']
• ❌ ALBERT (13 tokens): ['The', 'quick', 'brown', 'f', 'ox', 'jump', 's', 'over', 'the', 'la', 'zy', 'dog', '.']
• ❌ XLM-R (13 tokens): ['The', 'quick', 'brown', '', 'fox', 'jump', 's', 'over', 'the', 'la', 'zy', 'dog', '.']

💻 How to Use

1. Load the Model and Tokenizer

from transformers import AutoTokenizer, AlbertForMaskedLM

# Load from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("noson71/albert-khmer-codeswitch-64k")
model = AlbertForMaskedLM.from_pretrained("noson71/albert-khmer-codeswitch-64k")

# Test tokenization
text = "យើងកំពុងសិក្សាអំពី machine learning និង blockchain technology។"
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['យើង', 'កំពុង', 'សិក្សាអំពី', '▁', 'machine', '▁', 'learning', 'និង', '▁', 'blockchain', '▁', 'technology', '។']
# Note: '▁' represents a space prefix in the SentencePiece Metaspace decoder.

Citation

@misc{{noson71-albert-khmer-codeswitch-tokenizer,
  author = {{NOSON.AI}},
  title = {{ALBERT Khmer Code-Switching}},
  year = {{2026}},
  publisher = {{Hugging Face}},
  url = {{https://huggingface.co/noson71/albert-khmer-codeswitch}}
}}