BertMoji: Emoji Prediction with DeBERTa-v3

BertMoji predicts the most appropriate emoji for a given text message. Built on DeBERTa-v3-small, it classifies text into 250 emoji categories with 90.2% accuracy.

Model Description

• Base Model: microsoft/deberta-v3-small
• Task: Multi-class emoji classification (250 classes)
• Architecture: DeBERTa-v3 encoder + classification head
• Training: Fine-tuned on ~23,500 synthetic text-emoji pairs. The model was refined over several fine-tuning sessions and evaluations.

Performance

Quick Start

import torch
import torch.nn as nn
from transformers import AutoTokenizer, DebertaV2Model
import json

class BertmojiClassifier(nn.Module):
    def __init__(self, model_name, num_classes):
        super().__init__()
        self.encoder = DebertaV2Model.from_pretrained(model_name)
        hidden_size = self.encoder.config.hidden_size
        self.classifier = nn.Sequential(
            nn.Dropout(0.1),
            nn.Linear(hidden_size, hidden_size),
            nn.GELU(),
            nn.Dropout(0.1),
            nn.Linear(hidden_size, num_classes)
        )

    def forward(self, input_ids, attention_mask):
        outputs = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
        pooled = outputs.last_hidden_state[:, 0, :]
        return self.classifier(pooled)

# Load model
model_path = "your-username/bertmoji-deberta-v3-small"
tokenizer = AutoTokenizer.from_pretrained(model_path)

with open(f"{model_path}/emoji_mappings.json") as f:
    mappings = json.load(f)
id_to_emoji = {int(k): v for k, v in mappings['id_to_emoji'].items()}

model = BertmojiClassifier("microsoft/deberta-v3-small", len(id_to_emoji))
model.load_state_dict(torch.load(f"{model_path}/pytorch_model.bin", map_location="cpu"))
model.eval()

# Predict
def predict_emoji(text, top_k=3):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=96)
    with torch.no_grad():
        logits = model(inputs["input_ids"], inputs["attention_mask"])
        probs = torch.softmax(logits, dim=-1)
        top_probs, top_ids = probs.topk(top_k)
    return [(id_to_emoji[idx.item()], prob.item()) for idx, prob in zip(top_ids[0], top_probs[0])]

# Example
print(predict_emoji("This pizza is absolutely incredible"))
# Output: [('pizza_emoji', 0.98), ...]

Demo Examples

Training Details

Limitations

• Trained on synthetic English text; may not generalize to all languages or dialects
• Some emoji categories have limited training data
• Model reflects biases present in training data generation

License

MIT License

Citation

@misc{bertmoji2024,
  title={BertMoji: Emoji Prediction with DeBERTa-v3},
  author={Mitchell Currie},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/your-username/bertmoji-deberta-v3-small}
}