Wav2Vec2 XLS-R 300M Korean LM

Wav2Vec2 XLS-R 300M Korean LM is an automatic speech recognition model based on the XLS-R architecture. This model is a fine-tuned version of Wav2Vec2-XLS-R-300M on the Zeroth Korean dataset. A 5-gram Language model, trained on the Korean subset of Open Subtitles, was then subsequently added to this model.

This model was trained using HuggingFace's PyTorch framework and is part of the Robust Speech Challenge Event organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.

All necessary scripts used for training could be found in the Files and versions tab, as well as the Training metrics logged via Tensorboard.

As for the N-gram language model training, we followed the blog post tutorial provided by HuggingFace.

Model

Evaluation Results

The model achieves the following results on evaluation without a language model:

With the addition of the language model, it achieves the following results:

Training procedure

The training process did not involve the addition of a language model. The following results were simply lifted from the original automatic speech recognition model training.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7.5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2000
• num_epochs: 50.0
• mixed_precision_training: Native AMP

Training results

Disclaimer

Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.

Authors

Wav2Vec2 XLS-R 300M Korean LM was trained and evaluated by Wilson Wongso. All computation and development are done on OVH Cloud.

Framework versions

• Transformers 4.17.0.dev0
• Pytorch 1.10.2+cu102
• Datasets 1.18.2.dev0
• Tokenizers 0.10.3