kk-bert-small-bpe-kazqad
BERT-Small finetuned on KazQAD (Kazakh Reading Comprehension) starting from
Abzalbek89/kk-bert-small-bpe — pretrained with the bpe tokenizer.
Part of a comparative study of subword tokenizers for Kazakh small language models.
Test set results
| metric | value |
|---|---|
| Exact Match | 2.0273 |
| F1 (token-level) | 8.72 |
Training
| Base model | Abzalbek89/kk-bert-small-bpe |
| Dataset | IS2AI/KazQAD reading-comprehension v1.0 |
| Examples | 2,713 test, ~10K train |
| Epochs | 10 |
| Batch size | 12 |
| Max length | 384 (stride 128) |
| Learning rate | 3e-5 (warmup 10%, weight decay 0.01) |
| Precision | fp16 |
| Hardware | RTX A4000 |
| Time | 1.4 min |
| Seed | 42 |
Usage
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-bert-small-bpe-kazqad")
model = AutoModelForQuestionAnswering.from_pretrained("Abzalbek89/kk-bert-small-bpe-kazqad")
qa = pipeline("question-answering", model=model, tokenizer=tok)
qa(question="Кардиннің туған жылы қашан?",
context="Belcalis Marlenis Almánzar 1992 жылы 11 қазанда дүниеге келген...")
Companion artifacts
- Pretrained base:
Abzalbek89/kk-bert-small-bpe - KazNERD finetune:
Abzalbek89/kk-bert-small-bpe-kaznerd - Aggregate metrics:
Abzalbek89/kk-tokenizer-fertility-baseline