langdetect-distilbert-10lang-CE

Fine-tuned distilbert-base-multilingual-cased for language identification across 10 European languages, with a focus on short, chat-style input (as short as 3 characters).

Model Description

Standard language detection libraries perform poorly on short texts (F1 ≈ 0.62–0.87 on inputs under 30 characters). This model was fine-tuned with a chat-oriented training distribution to close that gap, achieving F1 = 0.9680 on short texts — 10–34 percentage points above all evaluated general-purpose tools.

Supported Languages

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="GaborMadarasz/langdetect-distilbert-10lang-CE",
    device=0,        # GPU; use -1 for CPU
)

classifier("Mikor érkezel haza?")
# [{'label': 'hu', 'score': 0.9997}]

classifier("Ok")
# [{'label': 'en', 'score': 0.9821}]

classifier("Bonjour!")
# [{'label': 'fr', 'score': 0.9993}]

Training Data

Training used a hybrid corpus combining three sources:

Chat-Oriented Length Distribution

Standard corpora are dominated by long formal sentences. To reflect real chat traffic, training samples were bucket-sampled by character length:

Europarl filled medium and long buckets. OpenSubtitles (film subtitles — naturally short and conversational) filled the short bucket where Europarl was insufficient.

Evaluation Results

Evaluated on a held-out test set of 150,000 samples (15,000 per language), stratified across the same three length buckets.

Overall Performance

Performance by Text Length (Benchmark – 3,000 samples)

This is the most critical dimension for chat use cases:

Per-Language F1 (Test Set)

Greek achieves near-perfect scores due to its unique script. Slovak and Czech show the lowest scores due to high lexical similarity — a known challenge across all language identification systems.

Comparison with Popular Tools (F1 Macro, 3,000 samples)

Benchmark hardware: NVIDIA RTX 3060 12 GB (model on GPU, other tools on CPU).
All tools evaluated on identical stratified samples from the same test set.

Training Configuration

TrainingArguments(
    num_train_epochs=5,
    per_device_train_batch_size=64,
    gradient_accumulation_steps=2,     # effective batch: 128
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    weight_decay=0.01,
    bf16=True,                         # BF16 — native on Ampere
    eval_strategy="epoch",
    metric_for_best_model="f1_macro",
    load_best_model_at_end=True,
)

Training was stopped at epoch 5 (early stopping patience = 3). Best checkpoint at epoch 4 (F1 macro = 0.9840 on validation set).

Limitations

• Language scope: Only the 10 listed languages are supported. Inputs in other languages will be misclassified as one of the 10.
• Slavic pairs: Slovak/Czech and Slovenian/Croatian confusion is inherent to the task due to lexical overlap. This is the primary error source (sk→cs: 1.4%, sl→hr: 1.5%).
• Script sensitivity: The model relies on character-level features. Transliterated or romanized text (e.g., Greek in Latin script) may degrade performance.
• Latency: CPU inference is ~5–10× slower than GPU. Consider ONNX export for CPU-only deployments.

Intended Use

This model is intended for:

• non-commercial / research only
• Language routing in multilingual chat applications
• Preprocessing pipelines for NLP tasks requiring language-specific models
• Content moderation systems operating across the listed 10 languages

It is not intended for:

• Detecting languages outside the 10 supported ones
• Processing very long documents (truncated at 256 tokens)
• Production use without human review in high-stakes decisions

Contact

gabor.madarasz@gmail.com

Citation

If you use this model, please cite the training corpora:

@inproceedings{koehn2005europarl,
  title     = {Europarl: A Parallel Corpus for Statistical Machine Translation},
  author    = {Koehn, Philipp},
  booktitle = {Proceedings of the MT Summit},
  year      = {2005}
}

@inproceedings{lison2016opensubtitles,
  title     = {OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles},
  author    = {Lison, Pierre and Tiedemann, J{\"o}rg},
  booktitle = {LREC},
  year      = {2016}
}

@misc{macocu2022,
  title  = {MaCoCu: Massive Collection and Curation of Monolingual and Bilingual Data},
  author = {Ba{\~n}{\'o}n, Marta and others},
  year   = {2022}
}