Fine-Tuned Sentiment Classification Model - SK_Morph_BLM (Universal multi-domain sentiment classification)

Model Overview

This model is a fine-tuned version of the SK_Morph_BLM model for the task of sentiment classification. It has been trained on datasets from multiple domains, including banking, social media, movie reviews, politics, and product reviews. Some of these datasets were originally in Czech and were machine-translated into Slovak using Google Cloud Translation.

Sentiment Labels

Each row in the dataset is labeled with one of the following sentiments:

• Negative (0)
• Neutral (1)
• Positive (2)

Dataset Details

The dataset used for fine-tuning comprises text records from various domains. Below are the details for each domain:

Banking Domain

• Source: Banking Dataset
• Description: Sentences from the annual reports of a commercial bank in Slovakia.
• Records per Class: 923
• Unique Words: 11,469
• Average Words per Record: 20.93
• Average Characters per Word: 142.41

Social Media Domain

• Source: Social Media Dataset
• Description: Data from posts on the Facebook social network.
• Records per Class: 1,991
• Unique Words: 114,549
• Average Words per Record: 9.24
• Average Characters per Word: 57.11

Movies Domain

• Source: Movies Dataset
• Description: Short movie reviews from ČSFD.
• Records per Class: 3,000
• Unique Words: 72,166
• Average Words per Record: 52.12
• Average Characters per Word: 330.92

Politics Domain

• Source: Politics Dataset
• Description: Sentences from Slovak parliamentary proceedings.
• Records per Class: 452
• Unique Words: 6,697
• Average Words per Record: 12.31
• Average Characters per Word: 85.22

Reviews Domain

• Source: Reviews Dataset
• Description: Product reviews from Mall.cz.
• Records per Class: 3,000
• Unique Words: 35,941
• Average Words per Record: 21.05
• Average Characters per Word: 137.33

Fine-Tuning Hyperparameters

The following hyperparameters were used during the fine-tuning process:

• Learning Rate: 1e-05
• Training Batch Size: 64
• Evaluation Batch Size: 64
• Seed: 42
• Optimizer: Adam (default)
• Number of Epochs: 15 (with early stopping)

Model Performance

The model was trained on data from all domains simultaneously and evaluated using stratified 10-fold cross-validation on each individual domain. The weighted F1-score, including the mean, minimum, maximum, and quartile values, is presented below for each domain:

Model Usage

This model is suitable for sentiment classification within the specific domains it was trained on, such as banking, social media, movies, politics, and product reviews. While it may not achieve high F1-scores across all text types, it is well-suited for a wide range of text within these trained domains. However, it may not generalize effectively to entirely different types of text outside these domains.

Example Usage

Below is an example of how to use the fine-tuned SK_Morph_BLM-sentiment-multidomain model in a Python script:

import torch
from transformers import RobertaForSequenceClassification, RobertaTokenizerFast
from huggingface_hub import snapshot_download

class SentimentClassifier:
    def __init__(self, tokenizer, model):
        self.model = RobertaForSequenceClassification.from_pretrained(model, num_labels=3)
        
        repo_path = snapshot_download(repo_id = tokenizer)
        sys.path.append(repo_path)

        # Import the custom tokenizer from the downloaded repository
        from SKMT_lib_v2.SKMT_BPE import SKMorfoTokenizer
        self.tokenizer = SKMorfoTokenizer()

    def tokenize_text(self, text):
        encoded_text = self.tokenizer.tokenize(text.lower(), max_length=256, return_tensors='pt', return_subword=False)
        return encoded_text

    def classify_text(self, encoded_text):
        with torch.no_grad():
            output = self.model(**encoded_text)
            logits = output.logits
            predicted_class = torch.argmax(logits, dim=1).item()
            probabilities = torch.softmax(logits, dim=1)
            class_probabilities = probabilities[0].tolist()
            predicted_class_text = self.model.config.id2label[predicted_class]
        return predicted_class, predicted_class_text, class_probabilities

# Instantiate the sentiment classifier with the specified tokenizer and model
classifier = SentimentClassifier(tokenizer="daviddrzik/SK_Morph_BLM", model="daviddrzik/SK_Morph_BLM-sentiment-multidomain")

# Example text to classify sentiment
text_to_classify = "Napriek zlepšeniu očakávaní je výhľad stále krehký."
print("Text to classify: " + text_to_classify + "\n")

# Tokenize the input text
encoded_text = classifier.tokenize_text(text_to_classify)

# Classify the sentiment of the tokenized text
predicted_class, predicted_class_text, logits = classifier.classify_text(encoded_text)

# Print the predicted class label and index
print(f"Predicted class: {predicted_class_text} ({predicted_class})")
# Print the probabilities for each class
print(f"Class probabilities: {logits}")

Here is the output when running the above example:

Text to classify: Napriek zlepšeniu očakávaní je výhľad stále krehký.

Predicted class: Positive (2)
Class probabilities: [0.04016311839222908, 0.4200247824192047, 0.5398120284080505]