Ferman intent classifier (XLM-R base, v2)
Classifies assistant commands into 34 intents in English, Arabic, Sorani
Kurdish and Badini Kurdish. Fine-tuned from xlm-roberta-base. Part of the
Ferman project: https://github.com/omarGH99/ferman-ai-assistant
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="OmarSY11/xlmr-intent-v2", top_k=3)
print(clf("set an alarm for six thirty am"))
print(clf("ئەلارمێکی نوێ بۆ شەشی بەیانی")) # Sorani
Training data
The v2 configuration of OmarSY11/ferman-nlu,
train split. English and Arabic derive from MASSIVE; Sorani and Badini are
machine-translated (see the dataset card for provenance and limits).
Training
| Base model | xlm-roberta-base |
| Max length | 64 tokens |
| Learning rate / batch size | 2e-5 / 32 |
| Epochs | up to 10, early stopping (patience 2) on validation macro-F1 |
| Warm-up / weight decay | 0.1 / 0.01 |
| Loss | cross-entropy, label smoothing 0.1, inverse-frequency class weights (power 0.5) |
| Augmentation | random word deletion or adjacent swap, 3 extra copies per Sorani and Badini training row |
| Seed | 42 |
The notebook is in the repo: ml/train_xlmr_intent_v2_runpod.ipynb.
Results
Held-out test split of the v2 data (4,648 rows).
| Language | Test rows | Accuracy | Macro-F1 |
|---|---|---|---|
| English | 1,954 | 0.887 | 0.847 |
| Arabic | 1,954 | 0.823 | 0.761 |
| Badini | 370 | 0.719 | 0.682 |
| Sorani | 370 | 0.697 | 0.647 |
| Overall | 4,648 | 0.832 | 0.790 |
Kurdish test sets are small, so those figures carry wide uncertainty. The
weakest intent is general_quirky, a catch-all class (F1 0.62 overall, 0.00 in
Sorani). Per-intent results and the confusion matrix are in the repo under
ml/results/.
Limitations
- Kurdish training and test data are machine-translated and informally reviewed by one person (native Badini, non-native Sorani).
- Sorani is the weakest language and is the one v2 did not improve over v1.
- 34 intents only. Out-of-domain input is rejected in the app by confidence and margin thresholds, not by the model itself.
- Not evaluated on real user speech or on other Kurdish varieties.
License and citation
Model weights: MIT, like the base model. Training data: CC BY 4.0, derived from MASSIVE (FitzGerald et al., 2022, arXiv:2204.08582), which must be credited. Please also cite https://github.com/omarGH99/ferman-ai-assistant.