DistilBERT hotel-review sentiment classifier

A full fine-tune of distilbert-base-uncased that labels an English hotel review as negative or positive. It is part of the krimits/hotel-review-nlp project. It was trained and tested on splits of Booking.com reviews that share no review text.

Result of this model

It was scored on its test set of 13,263 reviews (3,263 negative). That set shares no review text with training. It was scored once, with the epoch that did best on dev.

• Against Naive Bayes. On the same reviews, the gain is +0.0291 macro-F1, with a paired 95% interval of 0.0246 to 0.0335. Exact McNemar test:
  • 387 reviews only this model gets right;
  • 97 only Naive Bayes gets right;
  • p = 4.9 × 10⁻⁴².
• Training: 117,880 reviews. The best dev macro-F1 was 0.9615.

Related experiment, with a different checkpoint

A second model, with the same architecture and training recipe, was trained on another split of the same reviews.

• Training data: reviews written up to 9 March 2017.
• Result: 0.9491 macro-F1 (0.9450–0.9531) on reviews written from 26 May 2017 to 3 August 2017.
• What it means. That is a different checkpoint, trained on different data. It is not a measurement of this model, nor an estimate of it. It shows only that scores can be lower on reviews from another period.

Cost

• Size: 66,955,010 parameters, 255 MiB of weights.
• Speed. The timings come from the related checkpoint above. It has the same architecture and size as this model.
  • 57 ms per review (median, one review at a time, one thread of an Intel(R) Xeon(R) CPU @ 2.00GHz);
  • 254 reviews per second on a Tesla T4.
• Training time: 13.6 minutes on a Tesla T4.

Limits

• Data. English Booking.com reviews of European hotels, written from 4 August 2015 to 3 August 2017.
• Two classes. Reviews with both positive and negative text were left out of training and test.
• Test set. Each test set holds at most 10,000 positive reviews.
• One training seed (42). The interval covers the sampling of test reviews, not the randomness of training.

Training settings

distilbert-base-uncased, seed 42, 2 epochs, batch 32, maximum length 256, learning rate 2e-05, warm-up 6%, weight decay 0.01, fp16 training. The checkpoint kept is the epoch with the best dev macro-F1.

Provenance

• Training. Trained by notebook 10 of the project, from commit 994a103. Its metrics, logits and logs are committed. The CI recomputes its scores from those logits. The decision note compares it with Naive Bayes.
• Weights: model.safetensors SHA-256 f2a2bbc9a31c8f3b1d9bee46ec11bb3d8942e7e59c0ef4711c3cb2a6c1cf81c2.
• Check before publishing. An agreement check on 1,024 examples: the weights reproduced the saved test logits within an absolute tolerance of 0.001. The largest difference was 3.6e-06, and 1,024 of 1,024 predictions were equal.
• Publishing code: commit 21d535c of the project (scripts/publish_model.py).
• The previous model is kept under the tag legacy-split-v1 (commit 9fe2f7f). It was trained on an earlier split whose test set shared 170 texts with training.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("krimits/distilbert-hotel-reviews")
model = AutoModelForSequenceClassification.from_pretrained("krimits/distilbert-hotel-reviews")
inputs = tokenizer("The room was spotless and the staff was wonderful.", return_tensors="pt")
print(model.config.id2label[model(**inputs).logits.argmax(-1).item()])  # positive