Kurdish Sorani RoBERTa
A RoBERTa language model for Central Kurdish (Sorani), trained from scratch on a 1-million-sentence Kurdish corpus with a custom 32k Kurdish BPE tokenizer. It's a masked-language model meant as a base for fine-tuning on Kurdish tasks (classification, NER, embeddings, and so on).
It's the base model behind haryads/kurdish-sentence-embeddings.
Usage
from transformers import pipeline
fill = pipeline("fill-mask", model="haryads/kurdish-roberta")
fill("زمانی کوردی زۆر <mask> ە")
Or load it directly to fine-tune:
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("haryads/kurdish-roberta")
model = AutoModel.from_pretrained("haryads/kurdish-roberta")
Architecture
| Type | RoBERTa (masked LM) |
| Layers | 6 |
| Hidden size | 512 |
| Attention heads | 8 |
| Max sequence length | 128 |
| Vocab size | 32,000 |
| Parameters | ~35M |
Training
- Corpus: 1M Sorani sentences
- Tokenizer: custom 32k byte-level BPE for Kurdish
- Objective: masked language modeling (15% masking)
Intended use and limits
A base model, not an instruct or chat model. Best used fine-tuned for a specific Kurdish task. Sorani only, and the corpus is news-heavy, so very colloquial or domain-specific text may need in-domain fine-tuning.
License
Apache-2.0.