kk-bert-small-bpe
BERT-Small (~29.55M params) pretrained from scratch on the Kazakh corpus_clean corpus
using the bpe tokenizer.
Part of a comparative study of subword tokenizers (BPE, Unigram, morphology-aware) for Kazakh small language models.
Tokenizer
Abzalbek89/kk-tokenizer-bpe-32k — vocab 32K, trained on Kazakh news + literature.
Architecture
BERT-Small (Turc et al. 2019):
- 4 hidden layers
- 512 hidden size
- 8 attention heads
- 2048 intermediate size
- max position 512, sequence length 256
- ~29.55M parameters
Pretraining
| Steps | 10,000 |
| Batch size | 32 |
| Sequence length | 256 |
| Train blocks | 29,859 (≈7.6M tokens) |
| Optimizer | AdamW, lr 1e-4, warmup 10%, weight decay 0.01 |
| Precision | fp16 |
| Hardware | RTX A4000 (16 GB) |
| Wallclock | 17.2 min |
Evaluation
| metric | value |
|---|---|
| eval_loss | 7.6138 |
| eval_perplexity | 2025.952 |
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-bert-small-bpe")
model = AutoModelForMaskedLM.from_pretrained("Abzalbek89/kk-bert-small-bpe")
Citation
If you use this model, please cite the upcoming paper:
Tokenizer Optimization for Kazakh Small Language Models: A Comparative Study of BPE, Unigram, and Morphological Segmentation. (in prep.)
Companion artifacts
- Aggregate metrics:
Abzalbek89/kk-tokenizer-fertility-baseline - Sister models:
kk-bert-small-bpe,kk-bert-small-unigram,kk-bert-small-morph-bpe