albert-kh-en-codeswitch-32k

This model is a Khmer-English Code-Switching ALBERT based on seanghay/albert-khmer-small, further pre-trained on a bilingual corpus with explicit code-switching data.

The model excels at understanding and processing mixed Khmer-English text, named entity recognition, and question answering in code-switched contexts.

Model Details

Developed by: NOSON
Model type: ALBERT (A Lite BERT)
Language(s): Khmer (km), English (en)
License: Apache-2.0
Finetuned from model: seanghay/albert-khmer-small

Model Description

This model extends the original Khmer ALBERT with enhanced capabilities for:

Code-switching: Seamlessly handles mixed Khmer-English text.
Technical vocabulary: Better understanding of tech terms modern language.
Cultural preservation: Maintains high integrity of complex Khmer words.

Training Data

The model was trained on a carefully balanced corpus consisting of:

Khmer Wikipedia (2023-2024 dumps)
WikiText-103 (English)
MC4 (Khmer & English subsets)
Custom Code-Switching Corpus (30,000+ injected examples)

Training Procedure

Preprocessing

Custom 32,000 token SentencePiece Unigram tokenizer.
Character coverage: 1.0 (perfect for Khmer diacritics).
Forced vocabulary for technical terms and complex Khmer words.
Maximum sequence length: 512 tokens.

Evaluation Results

Lower TPC is better; higher percentages are better for integrity.

Metric	This Model (32K)	ALBERT Base (16K)	XLM-R Base (250K)
TPC (Khmer)	0.143	0.164	0.264
Tokenization Speed	0.3M tok/s	0.3M tok/s	0.4M tok/s

Complex Word Integrity Breakdown

A critical challenge in Khmer NLP is preventing the fragmentation of complex Sanskrit/Pali loanwords and compound words. Our custom tokenizer significantly outperforms baselines in keeping these words intact as single tokens:

Complex Word	Meaning	Our Model (32K)	ALBERT Base (16K)	XLM-R Base (250K)
ធម៌	Dharma	✅ 1 token	❌ 2 tokens	❌ 2 tokens
កម្ម	Karma	✅ 1 token	❌ 2 tokens	❌ 2 tokens
និព្វាន	Nirvana	✅ 1 token	❌ 4 tokens	❌ 3 tokens
អភិវឌ្ឍន៍	Development	❌ 2 tokens	❌ 2 tokens	❌ 2 tokens
សេដ្ឋកិច្ច	Economy	✅ 1 token	❌ 2 tokens	✅ 1 token
ប្រជាធិបតេយ្យ	Democracy	❌ 2 tokens	❌ 2 tokens	❌ 2 tokens
អធិបតេយ្យ	Sovereignty	❌ 2 tokens	❌ 2 tokens	❌ 5 tokens

MLM Performance Examples

Example 1: Technical Code-Switching

Input: យើងកំពុងសិក្សាអំពី [MASK] learning និង blockchain technology។
Top 5: machine, technology, data, software, learning

Example 2: Common Code-Switching

Input: ភាសាខ្មែរ is the official [MASK] of Cambodia.
Top 5: language, government, official, learning, data

Example 3: Pure Khmer

Input: ព្រះពុទ្ធសាសនាគឺជាសាសនាចម្បងក្នុង [MASK] កម្ពុជា។
Top 5: ព្រះរាជាណាចក្រ, ប្រទេសកម្ពុជា, សាសនា, ប្រវត្តិសាស្ត្រ, ពុទ្ធសាសនា

How to Use

Load the model

from transformers import AutoTokenizer, AlbertForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("noson71/albert-kh-en-codeswitch-32k")
model = AlbertForMaskedLM.from_pretrained("noson71/albert-kh-en-codeswitch-32k")

text = "យើងកំពុងសិក្សាអំពី machine learning និង blockchain technology។"
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['យើង', 'កំពុង', 'សិក្សាអំពី', '▁', 'machine', '▁', 'learning', 'និង', '▁', 'blockchain', '▁', 'technology', '។']

Masked Language Modeling (Inference)

import torch

text = "ភាសាខ្មែរ is the official [MASK] of Cambodia."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# Find the [MASK] token position and get the top prediction
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
predicted_token_id = logits[0, mask_token_index].argmax(axis=-1).item()

print("Prediction:", tokenizer.decode([predicted_token_id]))
# Expected output: 'language' or 'government'

Citation

@misc{noson71-khmer-en-codeswitch-albert,
  author = {NOSON},
  title = {ALBERT Khmer-English Code-Switching 32K},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/noson71/albert-kh-en-codeswitch-32k}
}