PhoBERT: Pre-trained language models for Vietnamese

Pre-trained PhoBERT models are the state-of-the-art language models for Vietnamese (Pho, i.e. "Phở", is a popular food in Vietnam):

• Two PhoBERT versions of "base" and "large" are the first public large-scale monolingual language models pre-trained for Vietnamese. PhoBERT pre-training approach is based on RoBERTa which optimizes the BERT pre-training procedure for more robust performance.
• PhoBERT outperforms previous monolingual and multilingual approaches, obtaining new state-of-the-art performances on four downstream Vietnamese NLP tasks of Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference.

The general architecture and experimental results of PhoBERT can be found in our EMNLP-2020 Findings paper:

@article{phobert,
title     = {{PhoBERT: Pre-trained language models for Vietnamese}},
author    = {Dat Quoc Nguyen and Anh Tuan Nguyen},
journal   = {Findings of EMNLP},
year      = {2020}
}

Please CITE our paper when PhoBERT is used to help produce published results or is incorporated into other software.

Users may also want to use BamiBERT, our new BERT-based pretrained language model for Vietnamese that addresses key limitations of PhoBERT. Trained from scratch on a 129 GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2,048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across eight Vietnamese benchmarks, it achieves the best performance on 11 of 15 metrics and the second-best performance on three others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization.