IMDB movie review sentiment (all-mpnet-base-v2 embedding + logistic regression)
This is the best lightweight configuration from the study Deep Learning Approaches for Sentiment Analysis of Engaged Audience in Motion Picture Realm: A Comparative Study of Traditional and Neural Network-Based Classification Models (Abir Hosen; supervisor Sultana Jahan Soheli, Department of ICE, Noakhali Science and Technology University). The study compared bag-of-words, TF-IDF and sentence-embedding features with logistic regression, SVM, naive Bayes, random forest, KNN and a neural network. Sentence embeddings gave the best results.
- Interactive demo (runs in the browser): https://huggingface.co/spaces/abir71/imdb-sentiment-demo
How it works
- Clean the review: strip HTML tags and lower-case it.
- Embed it with
sentence-transformers/all-mpnet-base-v2: mean pooling over the tokens, without the final normalisation, and at most 512 tokens (the library default is 384). - Classify with logistic regression (C = 10): it outputs the probability that the review is positive.
| File | Contents |
|---|---|
classifier.joblib |
scikit-learn LogisticRegression (scikit-learn 1.9.1) |
classifier.json |
the same weights as plain numbers (768 coefficients + intercept), used by the demo |
metadata.json |
settings, data counts and test results |
Usage
import joblib
from bs4 import BeautifulSoup
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer
st = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")
st[0].max_seq_length = 512
embedder = SentenceTransformer(modules=[st[0], st[1]]) # mean pooling, no normalisation
clf = joblib.load(hf_hub_download("abir71/imdb-sentiment-mpnet", "classifier.joblib"))
review = "The acting was superb and the story kept me hooked until the very end."
text = BeautifulSoup(review, "html.parser").get_text().lower()
print(clf.predict_proba(embedder.encode([text]))[0, 1]) # probability of positive
Evaluation
The data is the IMDB 50K movie reviews dataset (Maas et al., 2011), with 25,000 positive and 25,000 negative reviews. The 419 repeated reviews were removed first, so no test review also appears in training. The rest were split 80/20 (stratified, random_state 42) into 39,664 training and 9,917 test reviews.
| Classifier on the embedding | Accuracy | Precision | Recall | F1 | ROC-AUC |
|---|---|---|---|---|---|
| Logistic regression (this model) | 91.11 % | 90.39 % | 92.06 % | 91.22 % | 96.94 % |
| Neural network, 256 hidden units | 91.14 % | 90.65 % | 91.80 % | 91.22 % | 97.07 % |
The study's notebook reported 91.34 % for logistic regression and 91.98 % for an RBF-kernel SVM on the same embeddings, but without removing the repeated reviews. The SVM keeps tens of thousands of support vectors, so the far smaller logistic regression is published here.
The browser demo uses a 16-bit copy of the embedding model (Xenova/all-mpnet-base-v2, model_fp16.onnx). On 40
test reviews its probabilities were within 0.009 of the full-precision model, with identical labels. The 8-bit copy
was rejected because it changed 2 of the 40 labels.
Limitations
- Length and domain. It was trained on full-length IMDB reviews, so very short texts, tweets and non-movie reviews are less reliable.
- Two classes only. Neutral or mixed reviews are forced into positive or negative, usually with a probability close to 50 %.
- Sarcasm and irony are often missed.
- English only. Only the first 512 tokens of a review are read.
Citation
Hosen A. Deep learning approaches for sentiment analysis of engaged audience in motion picture realm: a comparative study of traditional and neural network-based classification models. Research report, Department of ICE, Noakhali Science and Technology University.
Data: Maas AL, Daly RE, Pham PT, Huang D, Ng AY, Potts C. Learning word vectors for sentiment analysis. ACL 2011.