albert-chinese-large-qa

Albert large QA model pretrained from baidu webqa and baidu dureader datasets.

Data source

• baidu webqa 1.0
• baidu dureader

Traing Method

We combined the two datasets together and created a new dataset in squad format, including 705139 samples for training and 69638 samples for validation. We finetune the model based on the albert chinese large model.

Hyperparams

• learning_rate 1e-5
• max_seq_length 512
• max_query_length 50
• max_answer_length 300
• doc_stride 256
• num_train_epochs 2
• warmup_steps 1000
• per_gpu_train_batch_size 8
• gradient_accumulation_steps 3
• n_gpu 2 (Nvidia Tesla P100)

Usage

from transformers import AutoModelForQuestionAnswering, BertTokenizer

model = AutoModelForQuestionAnswering.from_pretrained('wptoux/albert-chinese-large-qa')
tokenizer = BertTokenizer.from_pretrained('wptoux/albert-chinese-large-qa')

Important: use BertTokenizer

MoreInfo

Please visit https://github.com/wptoux/albert-chinese-large-webqa for details.