EmoMosaic-large

EmoMosaic-large is a well-competent model designed for classifying emotions in text, demonstrating strong performance across multiple domains. In fact, it outperforms recent state-of-the-art models on SemEval-2018 Task 1: Affect in Tweets and GoEmotions datasets. The model was developed as a part of my master's thesis.

Author

Author: Bc. Vít Tlustoš (tlustos.vit@gmail.com)

Supervisor: doc. Malik Aamir Saeed Ph.D

Thesis Text

Once the thesis has been defended, the text will be accessible at https://www.vut.cz/studenti/zav-prace/detail/153407. You are welcome to read it. Should you have any questions, please don't hesitate to contact me via the provided email.

Demo Application

As a part of the solution, we developed a Gradio application and deployed it on the Hugging Face Spaces platform. Once the thesis is made public, you can access it at: https://huggingface.co/spaces/vtlustos/EmoMosaic-space. This allows anyone to experiment with the models easily without requiring any technical skills or setup.

Models

To utilize these models within your application, first install all the necessary dependencies.

pip install torch
pip install transformers
pip install datasets

To utilize these models within your application, integrate following code and format your samples as context</s><s>sentence. The context is optional and represents sentences preceding the sentence to be classified, while sentence refers to the actual sentence undergoing classification. This example demonstrates how to use the EmoMosaic-base model. If you prefer to use its larger counterpart, replace vtlustos/EmoMosaic-base with vtlustos/EmoMosaic-large.

import torch
from transformers import RobertaTokenizer
from transformers import RobertaForSequenceClassification

# 1. initialize the model
tokenizer = RobertaTokenizer.from_pretrained(
    "vtlustos/EmoMosaic-base"
)
model = RobertaForSequenceClassification.from_pretrained(
    "vtlustos/EmoMosaic-base"
).to('cuda:0')

# 2. tokenize the sentences
tokens = tokenizer(
    [
        "All your work was lost when the computer crashed.</s><s>Oh my god. I spent a whole week on that."
    ],
    truncation=True,
    padding=True,
    return_tensors = "pt"
)    

# 3. make the prediction
with torch.no_grad():
    logits = model(
        tokens["input_ids"].to('cuda:0'), 
        tokens["attention_mask"].to('cuda:0')
    ).logits

# 4. convert to probabilities
preds = torch.sigmoid(logits)

print(preds)

After executing the code, you will receive a tensor with dimensions [S,E], where S represents the number of samples and E denotes the number of emotions. To associate individual probabilities with their respective emotions, use to the dictionary provided below:

ix2label = {
    "0": "admiration",
    "1": "amusement",
    "2": "anger",
    "3": "annoyance",
    "4": "anticipation",
    "5": "approval",
    "6": "caring",
    "7": "confusion",
    "8": "curiosity",
    "9": "desire",
    "10": "disappointment",
    "11": "disapproval",
    "12": "disgust",
    "13": "embarrassment",
    "14": "excitement",
    "15": "fear",
    "16": "gratitude",
    "17": "grief",
    "18": "happiness",
    "19": "joy",
    "20": "love",
    "21": "nervousness",
    "22": "optimism",
    "23": "pessimism",
    "24": "pride",
    "25": "realization",
    "26": "relief",
    "27": "remorse",
    "28": "sadness",
    "29": "surprise",
    "30": "trust"
}

Results

Here we present a brief overview of the results. For an in-depth analysis and discussion, please refer to the text of the thesis. The analysis covers model training, comparisons with other methods, assessments of performance at the level of individual categories, calibration, and qualitative evaluations across various scenarios.

SemEval-2018 Task 1: Affect in Tweets

Note: P and R denote precision and recall, respectively. Results are shown for our top-performing models measured on the test set of the SemEval-2018 Task 1: Affect in Tweets dataset.

GoEmotions

Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the GoEmotions dataset.

XED

Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the XED dataset.

DailyDialog

Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the DailyDialog dataset.

Per-Emotion Performance

EmoMosaic-base

EmoMosaic-large