Model Card: web-register-classification-multilingual-bge

A multilingual web register classifier covering 25 hierarchical labels from the CORE web register taxonomy. This model is the BGE-M3-based variant of TurkuNLP/web-register-classification-multilingual, fine-tuned from BAAI/bge-m3-retromae instead of XLM-RoBERTa-large.

Model Details

Model Description

• Developed by: TurkuNLP, University of Turku
• Model type: Sequence classification (XLMRobertaForSequenceClassification)
• Language(s): Trained on English, Finnish, French, Swedish and Turkish; expected to generalise to the other languages of the base model's multilingual pre-training (zero-shot results under Evaluation)
• License: Apache 2.0
• Fine-tuned from: BAAI/bge-m3-retromae

Model Sources

• Paper: Henriksson et al. (2024), Automatic register identification for the open web using multilingual deep learning, arXiv:2406.19892

Uses

Direct Use

Classify web documents by register/genre according to the CORE taxonomy. Given an input text, the model outputs probabilities for each of the 25 labels; the predicted register is the label (or labels) exceeding a chosen probability threshold.

Out-of-Scope Use

• Non-web text domains (e.g., historical documents, spoken transcripts) not represented in the CORE training data
• Languages not covered by the base model's multilingual training may yield unreliable predictions
• The model is not designed for fine-grained topical classification

Bias, Risks, and Limitations

• Automated predictions only: Register labels are probabilistic and can misclassify documents with mixed or ambiguous register profiles.
• Zero-shot transfer: Performance on languages beyond the 5 training languages is zero-shot and may be substantially lower.
• Threshold sensitivity: This model's probabilities run lower than those of the XLM-R-based TurkuNLP/web-register-classification-multilingual, so that model's thresholds don't transfer. Use the per-language thresholds in the Evaluation section.
• Web crawl data: The training corpus reflects the biases of web text (over-representation of certain topics, languages, and demographics).

Recommendations

Use the per-language threshold listed under Evaluation (Finnish 0.35, English 0.30, …). For a language not listed there, no tuned threshold exists; 0.30 is the most common value across the evaluated languages. Consult the CORE taxonomy documentation for label semantics before interpreting predictions.

How to Get Started with the Model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_name = "TurkuNLP/web-register-classification-multilingual-bge"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# This repository ships no tokenizer files; use the base model's tokenizer.
tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-m3-retromae")

text = "I just made the best pasta carbonara — here's the recipe!"
# The model was fine-tuned on inputs truncated to 512 tokens.
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits

probs = torch.sigmoid(logits[0])
threshold = 0.30  # English; use the per-language value from the Evaluation tables

id2label = model.config.id2label
predicted_labels = [id2label[i] for i, p in enumerate(probs) if p > threshold]
print("Predicted register(s):", predicted_labels)

The model predicts hierarchical labels: top-level labels (e.g. NA) and their subcategory labels (e.g. nb) may both appear. See the CORE taxonomy documentation for the full label hierarchy and meanings.

Register Labels

Training Details

Training Data

Multilingual CORE training data for English, Finnish, French, Swedish and Turkish, all 25 labels; this checkpoint is fold 6 of a 10-fold cross-validation setup.

Training Hyperparameters

From training_args.bin and training_metadata.json in this repository:

• Training regime: bfloat16
• Max sequence length: 512 tokens
• Batch size: 8 (one GPU, no gradient accumulation)
• Optimizer: AdamW (adamw_torch), learning rate 3e-05, linear schedule, no warmup, no weight decay
• Epochs: up to 30, evaluated each epoch; the best checkpoint by evaluation loss was kept, and training stopped after 23 epochs
• Seed: 42

Evaluation

Evaluation was performed on held-out test data (fold 6 of a cross-validation split) for each language. F1 (All) is micro-averaged F1 across all 25 labels; F1 (Main) is weighted F1 over the 9 top-level labels only (MT, LY, SP, ID, NA, HI, IN, OP, IP). The threshold column gives the per-language probability cutoff used for prediction. Because F1 (Main) is weighted, it is not directly comparable to the main-label F1 on the XLM-R model's card.

Trained languages

Zero-shot transfer (selected languages)

Performance on zero-shot languages varies considerably. The thresholds above are the per-language values recorded with each evaluation; the appropriate threshold for an unlisted language is unknown a priori.

Technical Specifications

Model Architecture

• Architecture: XLMRobertaForSequenceClassification
• Hidden size: 1024
• Attention heads: 16
• Hidden layers: 24
• Intermediate size: 4096
• Vocabulary size: 250,002
• Max position embeddings: 8,194
• Output classes: 25
• Precision: bfloat16

Compute Infrastructure

Trained on a single GPU on the Mahti supercomputer (CSC, Finland); the run's log directory names the Mahti GPU node it ran on.

Citation

If you use this model, please cite:

BibTeX:

@misc{henriksson2024automaticregisteridentification,
  title={Automatic register identification for the open web using multilingual deep learning},
  author={Erik Henriksson and Amanda Myntti and Anni Eskelinen and Selcen Erten-Johansson and Saara Hellstr{\"o}m and Veronika Laippala},
  year={2024},
  eprint={2406.19892},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2406.19892}
}

APA:

Henriksson, E., Myntti, A., Eskelinen, A., Erten-Johansson, S., Hellström, S., & Laippala, V. (2024). Automatic register identification for the open web using multilingual deep learning. arXiv:2406.19892.

Model Card Authors

Erik Henriksson, Tuomas Lundberg, Veronika Laippala (TurkuNLP, University of Turku)

Model Card Contacts

• Erik Henriksson — erikhenriksson (Hugging Face) | TurkuNLP, University of Turku
• Tuomas Lundberg — tuomaslundberg (Hugging Face) | TurkuNLP, University of Turku