DeBERTa V3 Base for Multilingual Quality Assessment

This is a fine-tuned version of the multilingual DeBERTa model (mdeberta) for assessing text quality across languages.

Model Details

• Architecture: mdeberta-v3-base-quality
• Task: Regression (Quality Assessment)
• Training Data: agentlans/tatoeba-english-translations dataset containing 44 900 English translations
• Input: Text in any of the supported languages by DeBERTa
• Output: Estimated quality score for text
  • higher values indicate better text

Performance

Root mean squared error (RMSE) on 20% held-out validation set: 0.5036

Training Data

The model was trained on agentlans/tatoeba-english-translations.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name="agentlans/mdeberta-v3-base-quality"

# Put model on GPU or else CPU
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

def quality(text):
    """Processes the text using the model and returns its logits.
    In this case, it's interpreted as the the combined quality score for that text."""
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True).to(device)
    with torch.no_grad():
        logits = model(**inputs).logits.squeeze().cpu()
    return logits.tolist()

quality("Your text here.")

Results

In this study, 10 English text samples of varying quality were generated and translated into Arabic, Chinese, French, Russian, and Spanish using Google Translate. This resulted in a total of 50 translated samples, which were subsequently analyzed by a trained classifier to predict their quality scores.

The following table presents the 10 original texts along with their translations:

The scatterplot below illustrates the predicted quality scores grouped by each text sample. Notably, the prediction scores exhibit low variability across different languages for the same text, indicating a consistent assessment of translation quality regardless of the target language.

This analysis highlights the effectiveness of using machine learning classifiers in evaluating textual quality across multiple languages.

Limitations

• Performance may vary for texts significantly different from the training data
• Output is based on statistical patterns and may not always align with human judgment
• Quality is assessed purely on textual features, not considering factors like subject familiarity or cultural context

Ethical Considerations

• Should not be used as the sole determinant of text suitability for specific audiences
• Results may reflect biases present in the training data sources
• Care should be taken when using these models in educational or publishing contexts