Language Detection Model

A BERT-based language detection model trained on hac541309/open-lid-dataset, which includes 121 million sentences across 200 languages. This model is optimized for fast and accurate language identification in text classification tasks.

Model Details

• Architecture: BertForSequenceClassification
• Hidden Size: 384
• Number of Layers: 4
• Attention Heads: 6
• Max Sequence Length: 512
• Dropout: 0.1
• Vocabulary Size: 50,257

Training Process

• Dataset:
  • Used the open-lid-dataset
  • Split into train (90%) and test (10%)
• Tokenizer: A custom BertTokenizerFast with special tokens for [UNK], [CLS], [SEP], [PAD], [MASK]
• Hyperparameters:
  • Learning Rate: 2e-5
  • Batch Size: 256 (training) / 512 (testing)
  • Epochs: 1
  • Scheduler: Cosine
• Trainer: Leveraged the Hugging Face Trainer API with Weights & Biases for logging

Data Augmentation

To improve model generalization and robustness, a new text augmentation strategy was introduced. This includes:

• Removing digits (random probability)
• Shuffling words to introduce variation
• Removing words selectively
• Adding random digits to simulate noise
• Modifying punctuation to handle different text formats

Impact of Augmentation

Adding these augmentations improved overall model performance, as seen in the latest evaluation results:

Evaluation

Updated Performance Metrics:

• Accuracy: 0.9733
• Precision: 0.9735
• Recall: 0.9733
• F1 Score: 0.9733

Detailed Evaluation (~12 millions texts)

Comparison with Previous Performance

After introducing text augmentations, the model's performance improved on the same evaluation dataset, with accuracy increasing from 0.9695 to 0.9733, along with similar improvements in average precision, recall, and F1 score.

Conclusion

The integration of new text augmentation techniques has led to a measurable improvement in model accuracy and robustness. These enhancements allow for better generalization across diverse language scripts, improving the model’s usability in real-world applications.

A detailed per-script classification report is also provided in the repository for further analysis.

---

How to Use

You can quickly load and run inference with this model using the Transformers pipeline:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("alexneakameni/language_detection")
model = AutoModelForSequenceClassification.from_pretrained("alexneakameni/language_detection")

language_detection = pipeline("text-classification", model=model, tokenizer=tokenizer)

text = "Hello world!"
predictions = language_detection(text)
print(predictions)

This will output the predicted language code or label with the corresponding confidence score.

---

Note: The model’s performance may vary depending on text length, language variety, and domain-specific vocabulary. Always validate results against your own datasets for critical applications.

For more information, see the repository documentation.

Thank you for using this model—feedback and contributions are welcome!