Bert Emotion Classifier

Fine-tuned BERT-base-uncased on GoEmotions for multi-label classification (28 emotions). This updated version includes improved Macro F1, ONNX support for efficient inference, and visualizations for better interpretability.

Model Details

• Architecture: BERT-base-uncased (110M parameters)
• Training Data: GoEmotions (58k Reddit comments, 28 emotions)
• Loss Function: Focal Loss (alpha=1, gamma=2)
• Optimizer: AdamW (lr=2e-5, weight_decay=0.01)
• Epochs: 5
• Batch Size: 16
• Max Length: 128
• Hardware: Kaggle P100 GPU (16GB)

Try It Out

For accurate predictions with optimized thresholds, use the Gradio demo. The demo now includes preprocessed text and the top 5 predicted emotions, in addition to thresholded predictions. Example predictions:

• Input: "I’m thrilled to win this award! 😄"
  • Output: excitement: 0.5836, joy: 0.5290
• Input: "This is so frustrating, nothing works. 😣"
  • Output: annoyance: 0.6147, anger: 0.4669
• Input: "I feel so sorry for what happened. 😢"
  • Output: sadness: 0.5321, remorse: 0.9107

Performance

• Micro F1: 0.6006 (optimized thresholds)
• Macro F1: 0.5390
• Precision: 0.5371
• Recall: 0.6812
• Hamming Loss: 0.0377
• Avg Positive Predictions: 1.4789

For a detailed evaluation, including class-wise accuracy, precision, recall, F1, MCC, support, and thresholds, along with visualizations, check out the Kaggle notebook.

Class-Wise Performance

The following table shows per-class metrics on the test set using optimized thresholds (see optimized_thresholds.json):

Visualizations

Class-Wise F1 Scores

Training Curves

Training Insights

The model was trained for 5 epochs with Focal Loss to handle class imbalance. Training and validation curves show consistent improvement:

• Training Loss decreased from 0.0429 to 0.0134.
• Validation Micro F1 peaked at 0.5874 (epoch 5).
• See the training curves plot above for details.

Usage

Quick Inference with inference.py (Recommended for PyTorch)

The easiest way to use the model with PyTorch is to programmatically fetch and use inference.py from the repository. The script handles all preprocessing, model loading, and inference for you.

Programmatic Download and Inference

Run the following Python script to download inference.py and make predictions:

# pip install transformers torch huggingface_hub emoji -q

from huggingface_hub import hf_hub_download
import importlib.util

# download inference script
path = hf_hub_download(repo_id="logasanjeev/bert-emotion-classifier", filename="inference.py")

# load module
spec = importlib.util.spec_from_file_location("inference", path)
inference = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inference)

# run prediction
text = "I’m thrilled to win this award! 😄"
result, processed = inference.predict_emotions(text)

print("Input:", text)
print("Processed:", processed)
print("Predicted Emotions:", result)

Expected Output:

Input: I’m thrilled to win this award! 😄
Processed: i’m thrilled to win this award ! grinning_face_with_smiling_eyes
Predicted Emotions:
excitement: 0.5836
joy: 0.5290

Alternative: Manual Download

If you prefer to download inference.py manually:

1. Install the required dependencies:

   pip install transformers torch huggingface_hub emoji
   

2. Download inference.py from the repository.
3. Use it in Python or via the command line.

Python Example:

from inference import predict_emotions

result, processed = predict_emotions("I’m thrilled to win this award! 😄")
print(f"Input: I’m thrilled to win this award! 😄")
print(f"Processed: {processed}")
print("Predicted Emotions:")
print(result)

Command-Line Example:

python inference.py "I’m thrilled to win this award! 😄"

Quick Inference with onnx_inference.py (Recommended for ONNX)

For faster and more efficient inference using ONNX, you can use onnx_inference.py. This script leverages ONNX Runtime for inference, which is typically more lightweight than PyTorch.

Programmatic Download and Inference

Run the following Python script to download onnx_inference.py and make predictions:

# pip install transformers torch huggingface_hub emoji -q

from huggingface_hub import hf_hub_download
import importlib.util

# download inference script
path = hf_hub_download(repo_id="logasanjeev/bert-emotion-classifier", filename="inference.py")

# load module
spec = importlib.util.spec_from_file_location("inference", path)
inference = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inference)

# run prediction
text = "I’m thrilled to win this award! 😄"
result, processed = inference.predict_emotions(text)

print("Input:", text)
print("Processed:", processed)
print("Predicted Emotions:", result)

Expected Output:

Input: I’m thrilled to win this award! 😄
Processed: i’m thrilled to win this award ! grinning_face_with_smiling_eyes
Predicted Emotions:
excitement: 0.5836
joy: 0.5290

Alternative: Manual Download

If you prefer to download onnx_inference.py manually:

1. Install the required dependencies:

   pip install transformers onnxruntime huggingface_hub emoji numpy
   

2. Download onnx_inference.py from the repository.
3. Use it in Python or via the command line.

Python Example:

from onnx_inference import predict_emotions

result, processed = predict_emotions("I’m thrilled to win this award! 😄")
print(f"Input: I’m thrilled to win this award! 😄")
print(f"Processed: {processed}")
print("Predicted Emotions:")
print(result)

Command-Line Example:

python onnx_inference.py "I’m thrilled to win this award! 😄"

Preprocessing

Before inference, preprocess text to match training conditions:

• Replace user mentions (u/username) with [USER].
• Replace subreddits (r/subreddit) with [SUBREDDIT].
• Replace URLs with [URL].
• Convert emojis to text using emoji.demojize (e.g., 😊 → smiling_face_with_smiling_eyes).
• Lowercase the text.

PyTorch Inference

from transformers import BertForSequenceClassification, BertTokenizer
import torch
import json
import requests
import re
import emoji

def preprocess_text(text):
    text = re.sub(r'u/\w+', '[USER]', text)
    text = re.sub(r'r/\w+', '[SUBREDDIT]', text)
    text = re.sub(r'http[s]?://\S+', '[URL]', text)
    text = emoji.demojize(text, delimiters=(" ", " "))
    text = text.lower()
    return text

repo_id = "logasanjeev/bert-emotion-classifier"
model = BertForSequenceClassification.from_pretrained(repo_id)
tokenizer = BertTokenizer.from_pretrained(repo_id)

thresholds_url = f"https://huggingface.co/{repo_id}/raw/main/optimized_thresholds.json"
thresholds_data = json.loads(requests.get(thresholds_url).text)
emotion_labels = thresholds_data["emotion_labels"]
thresholds = thresholds_data["thresholds"]

text = "I’m just chilling today."
processed_text = preprocess_text(text)
encodings = tokenizer(processed_text, padding='max_length', truncation=True, max_length=128, return_tensors='pt')
with torch.no_grad():
    logits = torch.sigmoid(model(**encodings).logits).numpy()[0]
predictions = [(emotion_labels[i], round(logit, 4)) for i, (logit, thresh) in enumerate(zip(logits, thresholds)) if logit >= thresh]
predictions = sorted(predictions, key=lambda x: x[1], reverse=True)
print(predictions)
# Output: [('neutral', 0.8147)]

ONNX Inference

For a simplified ONNX inference experience, use onnx_inference.py as shown above. Alternatively, you can use the manual approach below:

import onnxruntime as ort
import numpy as np

onnx_url = f"https://huggingface.co/{repo_id}/raw/main/model.onnx"
with open("model.onnx", "wb") as f:
    f.write(requests.get(onnx_url).content)

text = "I’m thrilled to win this award! 😄"
processed_text = preprocess_text(text)
encodings = tokenizer(processed_text, padding='max_length', truncation=True, max_length=128, return_tensors='np')
session = ort.InferenceSession("model.onnx")
inputs = {
    'input_ids': encodings['input_ids'].astype(np.int64),
    'attention_mask': encodings['attention_mask'].astype(np.int64)
}
logits = session.run(None, inputs)[0][0]
logits = 1 / (1 + np.exp(-logits))  # Sigmoid
predictions = [(emotion_labels[i], round(logit, 4)) for i, (logit, thresh) in enumerate(zip(logits, thresholds)) if logit >= thresh]
predictions = sorted(predictions, key=lambda x: x[1], reverse=True)
print(predictions)
# Output: [('excitement', 0.5836), ('joy', 0.5290)]

License

This model is licensed under the MIT License. See LICENSE for details.

Usage Notes

• The model performs best on Reddit-style comments with similar preprocessing.
• Rare emotions (e.g., grief, support=6) have lower F1 scores due to limited data.
• ONNX inference requires onnxruntime and compatible hardware (opset 14).