Binary Product Review Sentiment with DistilBERT

Model description

This 67M parameter model is a fine-tuned version of bert-base-uncased trained on the Cleanlab Amazon Reviews dataset. It achieves the following results on the evaluation set: Loss: 0.1785 Accuracy: 0.9440

Launch it locally

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="profautorino/product-review-sentiment-bert-base-uncased",
)

print(classifier("The headphones sound great and are comfortable to wear."))

Batch inference

For batch inference, pass a list of review strings:

reviews = [
    "This product exceeded my expectations!",
    "It broke after one day and was a waste of money.",
]
predictions = classifier(reviews, truncation=True)

for review, prediction in zip(reviews, predictions):
    print(f"{prediction['label']} ({prediction['score']:.3f}): {review}")

Intended uses & limitations

Intended to be used for batch sentiment analysis of product reviews in testing. Use in production is not advised. Use at your own risk.

Training and evaluation data

See Cleanlab/amazon-reviews for details: https://huggingface.co/datasets/Cleanlab/amazon-reviews

Training procedure

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 313
• eval_batch_size: 63
• seed: 42
• optimizer: AdamW
• num_epochs: 2

Training results

• Epoch 1 average training loss: 0.3395
• Epoch 2 average training loss: 0.2839

Quantization results

After quantizing the model from F32 to F16, this model maintains accuracy and produces the following evaluation results:
Split: test; evaluated reviews: 1000
Accuracy: 0.9440

          precision    recall  f1-score   support

negative       0.99      0.89      0.93       447
positive       0.91      0.99      0.95       553

accuracy                           0.94      1000
macro avg      0.95      0.94      0.94      1000
weighted avg   0.95      0.94      0.94      1000

Confusion matrix (rows=true, columns=predicted): [[396 51] [ 5 548]]