DistilBERT: Airbnb listing quality from review text

Fine-tuned distilbert-base-uncased that reads one guest review and predicts whether it comes from a below-par (overall rating <= 4.6) or **top-tier** (rating >= 4.9) Dallas Airbnb listing.

Code, data pipeline and tests: https://github.com/sgaind-data/Airbnb_Review_Scores_ML

Results (test set, listings never seen in training)

Model	Accuracy	Macro F1	F1 below-par	F1 top-tier
VADER (default 0.05)	0.547	0.436	0.187	0.686
VADER (tuned 0.88)	0.608	0.608	0.597	0.618
DistilBERT (fine-tuned)	0.704	0.704	0.700	0.708

Test set: 3,139 reviews from unseen listings.

Labels

0 = below_par, 1 = top_tier. Labels come from the listing's overall rating, not the individual review.

Training data

Inside Airbnb, Dallas TX, June 2024 scrape (CC BY 4.0, insideairbnb.com). 15,694 English reviews, balanced 50/50, max 25 reviews per listing, split 70/10/20 by listing so no listing appears in two splits. 3 epochs, lr 2e-5, batch 32, max length 256, free Colab T4 (about 3 minutes).

Limitations

Labels are listing-level: a happy guest at a 4.5 listing counts as below-par, which caps accuracy near 70%.
Dallas only, English only, reviews from 2010-2024.
Most confident errors are reviews whose sentiment disagrees with the listing average; the model mainly detects complaints (cleanliness, price and fees).

Usage

from transformers import pipeline
clf = pipeline("text-classification", model="sgaind-dataai/distilbert-airbnb-listing-quality")
clf("Great location but the sheets were dirty and parking was expensive.")

Loading…