Albanian ALBERT model pretrained on around 16GB of text (I used uonlp/CulturaX's sq configuration) and 1.1 million training steps, using only the masked language modelling task. Trained on a TPU v4-32 pod, made possible through the Google TPU Research Cloud.
Hyperparameters:
- Optimizer: LAMB
- LR: 0.0006
- \( \beta_1 \): 0.9
- \( \beta_2 \): 0.999
- \( \epsilon \): 1e-8
- Batch size: 1024
- Num. steps: 1.1 million
- dtype: bfloat16
- max. seq. length: 512
Evaluation
All downstream numbers below come from fine-tuning this model (albert-base-v2-sq) end to end on each task and, for context, fine-tuning mBERT (bert-base-multilingual-cased) and XLM-R-base (xlm-roberta-base) with exactly the same procedure (XLM-R-large is added for the tagging tasks). The aim is a like-for-like comparison, not state-of-the-art numbers.
Protocol:
- Model selection on validation data only. For each (task, model) the learning rate is picked by mean validation score over seeds, the checkpoint/epoch is picked on validation, and the test set is scored once at that epoch. Learning-rate grids: 1e-5/2e-5/5e-5 for classification, 2e-5/5e-5 (1e-5/2e-5 for XLM-R-large) for tagging, 2e-5/5e-5/1e-4 for retrieval.
- Multiple seeds. Classification and tagging use 5 seeds (42-46) and report mean ± std; the std includes initialisation and data-order noise, and for POS also the random train/dev/test split (the split is identical across models for a given seed). Retrieval was swept at one seed and repeated at three seeds for the chosen learning rate.
- Uncertainty. Bootstrap 95% CIs over test examples and paired bootstraps between models (same examples) are in the
REPORT*.mdfiles. Where a difference is not clearly larger than the noise, this is stated. - Same classifier head for all models. Mean pooling + dropout + linear layer, so that ALBERT's untrained pooler is not compared with mBERT's pretrained one. Max length 256 (classification), batch size 16 (8 for POS), 10 epochs (5 for MASSIVE and WikiANN, 15 for AlbNER/POS), bf16.
- Caveat on the pretraining. This checkpoint was trained with masked language modelling only (no NSP/SOP), which may affect sentence-level tasks; I could not test this.
Classification Tasks
Test accuracy, mean ± std over 5 seeds. Macro-F1 is given where it differs from accuracy.
| Task | Learning Rate | Number of epochs | albert-sq | mBERT | XLM-R-base |
|---|---|---|---|---|---|
| AlbMoRe[1] (sentiment) | 5e-05 | 10 | 0.957 ± 0.008 | 0.938 ± 0.015 | 0.976 ± 0.007 |
| MASSIVE-sq[4] (intent, 60 classes) | 5e-05 | 5 | 0.860 ± 0.002 (F1 0.803) | 0.851 ± 0.005 (F1 0.811) | 0.863 ± 0.006 (F1 0.816) |
| SIB-200 als_Latn[5] (topic, 99 test sentences) | 5e-05 | 10 | 0.828 ± 0.021 | 0.857 ± 0.013 | 0.873 ± 0.017 |
Notes:
- AlbMoRe: albert-sq is ahead of mBERT (paired CI [-0.007, +0.047], not significant) and behind XLM-R-base (CI [-0.038, -0.001]). MASSIVE: albert-sq is ahead of mBERT in accuracy (+0.9 pt, CI [+0.2, +1.6]) but not in macro-F1, and indistinguishable from XLM-R-base. SIB-200's test set is tiny, so none of the gaps is significant.
Sequence Labelling Tasks
Entity-level F1 (NER) and token accuracy (POS), test, mean ± std over 5 seeds. Span F1 is computed with a BIO span matcher (exact span and type).
| Task | albert-sq | mBERT | XLM-R-base | XLM-R-large |
|---|---|---|---|---|
| AlbNER[2] (official splits) | 0.794 ± 0.015 | 0.777 ± 0.020 | 0.793 ± 0.013 | 0.810 ± 0.025 |
| WikiANN-sq[3] | 0.912 ± 0.004 | 0.932 ± 0.002 | 0.931 ± 0.005 | 0.935 ± 0.003 |
| POS, Gheg-GPS[7] (trained on GPS) | 0.936 ± 0.004 | 0.927 ± 0.005 | 0.940 ± 0.003 | 0.950 ± 0.001 |
| POS, Gheg-GPS (trained on TSA+GPS) | 0.934 ± 0.004 | 0.930 ± 0.005 | 0.943 ± 0.003 | 0.952 ± 0.004 |
| POS, Albanian-TSA[6] (trained on TSA+GPS) | 0.940 ± 0.024 | 0.940 ± 0.021 | 0.949 ± 0.020 | 0.950 ± 0.015 |
| POS, Albanian-TSA (trained on TSA) | 0.938 ± 0.021 | 0.937 ± 0.015 | 0.926 ± 0.017 | 0.900 ± 0.013 |
Notes:
- albert-sq matches XLM-R-base on AlbNER (+0.0007, CI [-0.022, +0.023]) and is ahead of mBERT in 4 of 5 seeds, but the paired CI includes 0. On WikiANN-sq it is clearly worse than the other three (about -2 pt, 0/5 seeds ahead).
- On GPS POS the order is stable: XLM-R-large > XLM-R-base > albert-sq > mBERT. The TSA test set has only about 9 sentences, so all TSA differences are within noise; XLM-R-large's low TSA-only score reflects overfitting on 42 training sentences.
- POS bootstrap intervals resample only 5 seeds and are rough; the per-seed win counts in
REPORT_pos.mdare more informative.
Retrieval Tasks
Dense bi-encoder (mean pooling, L2-normalised, in-batch negatives with temperature 0.05, 3 epochs, batch 64, queries ≤ 64 / passages ≤ 256 tokens), fine-tuned on Albanian MS MARCO[8] (kzifla/albanian-msmarco). That dataset has no training qrels, so the 55,578 dev queries with qrels were split 49,578 train / 1,000 val / 5,000 test (split seed 42). Metric: MRR@10 on the 5,000 test queries, computed directly (no query/passage-id filtering).
| Eval | albert-sq | mBERT | XLM-R-base |
|---|---|---|---|
| MS MARCO-sq, 206k-passage subset, seed 42 (val-selected LR) | 0.410 (1e-4) | 0.499 (2e-5) | 0.516 (5e-5) |
| MS MARCO-sq, full 8.84M-passage corpus, same checkpoints | 0.118 | 0.151 | 0.154 |
| WebFAQ-sq[9], MS MARCO-fine-tuned model, 208 queries (transfer) | 0.751 | 0.760 | 0.828 |
Notes:
- The subset (val+test positives plus 200k random distractors) inflates absolute scores; use the full-corpus row for absolute numbers and the subset row only for relative comparison.
- For mBERT and XLM-R-base, 3 seeds at the chosen LR give 0.500 ± 0.010 and 0.516 ± 0.000 on the subset. For albert-sq, seed 42 and 43 land at about 0.41, but the 1e-4 run with seed 44 diverged at the peak learning rate (validation MRR 0), so no stable multi-seed mean is claimed. Full-corpus numbers are seed 42 only.
- WebFAQ-sq CIs are about ±0.05; albert-sq and mBERT are indistinguishable there. Zero-shot (before fine-tuning) retrieval scores did not predict the fine-tuned ranking and are not used here.
Summary
Averaged over tasks, albert-sq is at or near mBERT and generally below XLM-R-base: it is competitive on AlbNER, POS and MASSIVE, behind on WikiANN, SIB-200 and AlbMoRe, and clearly behind on MS MARCO retrieval. Many of the gaps are within seed noise on the smaller test sets; see the confidence intervals above.
References
[1] Çano, E. (2023). Albmore: A corpus of movie reviews for sentiment analysis in albanian. arXiv preprint arXiv:2306.08526.
[2] Çano, E. (2023). AlbNER: A Corpus for Named Entity Recognition in Albanian. arXiv preprint arXiv:2309.08741.
[3] Pan, X., Zhang, B., May, J., Nothman, J., Knight, K., & Ji, H. (2017). Cross-lingual Name Tagging and Linking for 282 Languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1946–1958, Vancouver, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/P17-1178. Albanian splits as distributed by Rahimi, A., Li, Y., & Cohn, T. (2019), Massively Multilingual Transfer for NER, ACL 2019 (P19-1015).
[4] FitzGerald, J., Hench, C., Peris, C., Mackie, S., Rottmann, K., Sanchez, A., Nash, A., Urbach, L., Kakarala, V., Singh, R., Ranganath, S., Crist, L., Britan, M., Leeuwis, W., Tur, G., & Natarajan, P. (2023). MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4277–4302, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-long.235
[5] Adelani, D. I., Liu, H., Shen, X., Vassilyev, N., Alabi, J. O., Mao, Y., Gao, H., & Lee, E.-S. A. (2024). SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 226–245, St. Julian's, Malta. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-long.14
[6] Universal Dependencies Albanian-TSA treebank (Toska; supervised by J. Nivre), UD v2.6+. https://universaldependencies.org/treebanks/sq_tsa/
[7] Universal Dependencies Gheg-GPS treebank (Ebert, Islamaj, Kuqi, Sonnenhauser, Widmer, Plamada), UD v2.11. https://universaldependencies.org/treebanks/aln_gps/
[8] Nguyen, T., et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268. Albanian translation: https://huggingface.co/datasets/kzifla/albanian-msmarco
[9] Dinzinger, M., Caspari, L., Ghosh Dastidar, K., Mitrović, J., & Granitzer, M. (2025). WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '25), pp. 3802–3811, Padua, Italy. ACM. https://doi.org/10.1145/3726302.3731934