Calibrated Review Triage & Cost-Optimal Routing (RoBERTa-large)
I fine-tuned FacebookAI/roberta-large for three-class English review classification—Negative (1–2 stars), Neutral (3 stars), and Positive (4–5 stars)—and paired it with out-of-fold probability calibration, selective human-review routing under capacity constraints, a DKK review-cost versus missed-error trade-off framework, and exploratory sentence-level four-star issue triage.
Interactive Space · Public evaluation set
Business use
- Who would use it & decision: Customer support and reputation management teams handling large volumes of reviews can use this model's confidence gap to decide what share of each batch to send to human review versus route automatically.
- Review cost per item: Using Statistics Denmark StatBank table
LONS20(2024 standardized hourly earnings for 42 Customer services clerks, all sectors: 246.15 DKK/hour) and a review time of 1 minute per review (assumption, 60 reviews/hour), reviewing one item costs 4.10 DKK (≈ EUR 0.55 / USD 0.61) (unrounded 4.1025 DKK). - Missed-error loss: Setting the loss of one unreviewed auto-route error to 25 DKK (≈ EUR 3.34 / USD 3.74) (assumption, representing ~6 minutes of downstream manual rework when a misclassified review is auto-routed).
- Recommended review share: Across the 0%–30% OOF routing table below, 20% human review (
100reviewed items and11missed auto-route errors per 500 reviews) minimizes total cost at 685.00 DKK (≈ EUR 91.64 / USD 102.57) at rounded4.10 DKK/item (or 685.25 DKK (≈ EUR 91.67 / USD 102.60) at unrounded4.1025 DKK/item, consisting of410.25 DKKreview cost +275.00 DKKerror loss), vs. 975.00 DKK (≈ EUR 130.43 / USD 145.99) at 0% review; because the optimal share changes when the loss assumption changes, you can adjust both inputs in the Review cost trade-off tab of the Interactive Space. Converted at ECB reference rates of 2026-11-09: 1 EUR = 7.4753 DKK = 1.1193 USD. (ECB euro foreign exchange reference rates) - Sensitivity: The cheapest review share depends on the missed-error loss assumption. With review cost at 4.10 DKK per item, it is 5% when a missed error costs about 7.6–20.8 DKK, 20% at about 20.8–31.5 DKK, and 25% at about 31.5–63.6 DKK; 10% and 15% are never the cheapest tier. The 25 DKK above is one example, not a finding (DKK; convert at the rates above).
- Boundaries: Evaluated on public Yelp data only, not deployed in any live business, assumes human review is 100% accurate (reviewer accuracy is unmeasured), and uses approximate figures from a 500-review sample.
Original training-run evaluation: 40,000 reviews
My original training run achieved 93.76% accuracy and 0.9389 weighted F1 on the full 40,000-review held-out test set. The separate, publicly reproducible 500-review evaluation below achieves 92.40% accuracy and 0.9201 weighted F1 with raw predictions and provides the complete calibration and triage benchmark.
| Class / metric | Precision | Recall | F1 |
|---|---|---|---|
| Negative | 0.9393 | 0.9261 | 0.9327 |
| Neutral | 0.6934 | 0.7661 | 0.7280 |
| Positive | 0.9789 | 0.9674 | 0.9731 |
| Overall | 0.9407 (weighted) | 0.9376 (accuracy) | 0.9389 (weighted) |
Labels
| ID | Meaning | Rating bin |
|---|---|---|
| 0 | Negative | 1–2 stars |
| 1 | Neutral | 3 stars |
| 2 | Positive | 4–5 stars |
Inference
from transformers import pipeline
classifier = pipeline(
"text-classification", model="Zhoyang/review-triage-roberta-large",
revision="2e39ef29d577aa84cfe70596020eb40e708b88a0", top_k=None,
truncation=True, max_length=512,
)
scores = classifier("The food was okay, but the service was slow.")
These are raw model class scores. They do not identify five-star probabilities, churn risk, or the value of an intervention. The model parameters are unchanged by post-hoc calibration.
Training details
I trained the 355M-parameter FacebookAI/roberta-large model on an NVIDIA RTX A4000 via Paperspace Gradient. The Yelp Academic dataset contained 499,229 reviews, with 239,870 selected for training and 40,000 held out for testing. I used a 512-token limit and class-weighted loss: Negative 3.0 / Neutral 8.0 / Positive 1.5.
Across seven controlled iterations from BERT-base, increasing the token limit from 256 to 512 produced the largest single-step weighted-F1 gain (+0.012).
Iteration History
| Version | Key Change | Weighted F1 | Neutral F1 |
|---|---|---|---|
| v1 | BERT-base baseline | 0.881 | 0.514 |
| v3 | RoBERTa-base | 0.892 | 0.573 |
| v4 | max_len 256→512 |
0.904 | 0.604 |
| v6 | RoBERTa-large | 0.912 | 0.612 |
| Final | EDA-driven class weights | 0.9389 | 0.728 |
Public 500-row benchmark
I evaluated all 500 reviews in Zhoyang/review-triage-500-eval. Raw predictions achieve 92.40% accuracy and 0.9201 weighted F1. OOF (out-of-fold) calibration results use five-fold cross-fitting. Temperature scaling improves NLL and ECE on this sample; temperature plus bias does not improve classification accuracy.
| Policy | Accuracy | Weighted F1 | Macro F1 | NLL | Brier | ECE |
|---|---|---|---|---|---|---|
| Raw softmax | 92.40% | 0.9201 | 0.8345 | 0.2372 | 0.1200 | 0.0364 |
| Inverse-weight sensitivity baseline | 92.80% | 0.9212 | 0.8300 | 0.2719 | 0.1230 | 0.0494 |
| OOF temperature | 92.40% | 0.9201 | 0.8345 | 0.2184 | 0.1168 | 0.0147 |
| OOF temperature + bias | 92.20% | 0.9176 | 0.8288 | 0.2236 | 0.1188 | 0.0159 |
Raw per-class performance on these 500 rows
| Class | Support | Precision | Recall | F1 |
|---|---|---|---|---|
| Negative | 115 | 0.9310 | 0.9391 | 0.9351 |
| Neutral | 50 | 0.6923 | 0.5400 | 0.6067 |
| Positive | 335 | 0.9478 | 0.9761 | 0.9618 |
Raw confusion matrix
Rows are true three-class labels; columns are predictions.
| True / predicted | Negative | Neutral | Positive |
|---|---|---|---|
| Negative | 108 | 6 | 1 |
| Neutral | 6 | 27 | 17 |
| Positive | 2 | 6 | 327 |
Error breakdown by original rating
| Original stars | n | Raw errors | Raw error rate | OOF temp+bias error rate |
|---|---|---|---|---|
| 1 | 85 | 1 | 1.18% | 1.18% |
| 2 | 30 | 6 | 20.00% | 20.00% |
| 3 | 50 | 23 | 46.00% | 48.00% |
| 4 | 85 | 5 | 5.88% | 5.88% |
| 5 | 250 | 3 | 1.20% | 1.20% |
Errors are concentrated at the 2–3-star boundary: raw error rates are 20.00% and 46.00%, compared with 1.18% for 1-star and 1.20% for 5-star reviews. These are errors against the three-class rating target, not five-star prediction errors.
Human review capacity and held-out threshold routing
I route low-margin predictions to review, using the gap between the two highest class probabilities. Each fold learns its calibrated margin thresholds on the other four folds, then applies them to held-out reviews. The reference budget therefore need not equal the observed flagged share.
| Reference budget | OOF flagged share | Auto-route n | Auto-route error rate | Errors in review queue |
|---|---|---|---|---|
| 0% | 0.0% | 500 | 7.80% | 0 |
| 5% | 4.8% | 476 | 5.46% | 13 |
| 10% | 9.4% | 453 | 4.86% | 17 |
| 15% | 15.2% | 424 | 3.77% | 23 |
| 20% | 20.0% | 400 | 2.75% | 28 |
| 25% | 24.6% | 377 | 2.12% | 31 |
| 30% | 30.8% | 346 | 1.73% | 33 |
The Interactive Space includes a Review cost trade-off tab built directly on this 500-review OOF threshold routing table (0% to 30% in 5% steps, where human-reviewed counts $500 - \text{Auto-route } n$ are 0, 24, 47, 76, 100, 123, 154 and remaining auto-route errors are 39, 26, 22, 16, 11, 8, 6). Given two user-supplied hypothetical unit weights in the same unit (cost to review one item and loss from one missed auto-route error), it computes human review count, remaining auto-route errors, human review cost, missed-error loss, and total cost at each tier, highlights the lowest-total tier, and reports how many additional reviews must be inspected to catch one additional error (1.85 at 0%→5%, 5.75 at 5%→10%, 4.83 at 10%→15%, 4.80 at 15%→20%, 7.67 at 20%→25%, and 15.50 at 25%→30%). This calculation rests on two explicit assumptions: (1) human review is treated as 100% accurate (actual reviewer accuracy has not been measured), and (2) the sample contains only 500 reviews, so all figures are approximate. It uses hypothetical unit weights only and does not claim real business costs, customer outcomes, or production deployment.
Retrospective batch ranking with an exact review budget
For a fixed batch, ranking by ascending top-two margin selects exactly the requested number of reviews. This answers a different capacity question from reference-threshold routing. Both tables report errors only among automatically routed reviews; reviewer corrections are not measured.
| Exact review budget | Review n | Raw auto-route error rate | OOF temp+bias auto-route error rate |
|---|---|---|---|
| 0% | 0 | 7.60% | 7.80% |
| 5% | 25 | 5.26% | 5.26% |
| 10% | 50 | 4.44% | 4.67% |
| 15% | 75 | 3.76% | 4.00% |
| 20% | 100 | 2.50% | 3.00% |
| 25% | 125 | 2.13% | 2.13% |
| 30% | 150 | 1.71% | 1.71% |
Final reference calibration parameters
{
"temperature": 1.54,
"logit_bias": [
-0.2700000000000001,
-0.5399999999999999,
0.0
]
}
I fit these reference parameters on all 500 reviews for the Space. Their performance on new reviews has not been independently evaluated; the OOF results above evaluate the cross-fitting procedure. The Space defaults to raw scores, with experimental calibration available as an option.
Exploratory four-star review triage
This experiment tests whether the existing three-class review classifier can help prioritize four-star reviews for human inspection of issue mentions.
In a development pool of 85 four-star Yelp reviews, one AI annotator provisionally labeled 20 reviews as containing concrete operational or product issues (Target B) and 36 as containing any explicit complaint or caveat (Target A). At a budget of 20 reviews, ranking by the lowest standalone sentence Positive score retrieved 13 labeled Target B positives, compared with 10 for whole-review scoring and 10 for a fixed negative-word lexicon. A lexicon expanded during development retrieved 12, and ranking by sentence count alone retrieved 9. The sentence method's precision and recall on Target B were both 65% at $K = 20$ on this labeled pool.
| Ranking policy (85 four-star reviews) | Role | $K=10$ Target B (A) | $K=20$ Target B (A) | $K=30$ Target B (A) |
|---|---|---|---|---|
Sentence minimum P(Pos) ($\ge 12$ chars) |
Experimental candidate | 8 (9) | 13 (18) | 16 (24) |
Sentence minimum P(Pos) (unfiltered) |
Sensitivity check | 7 (9) | 13 (17) | 17 (24) |
Whole-review P(Pos) |
Comparison baseline | 6 (9) | 10 (17) | 10 (22) |
| Fixed negative-word lexicon | Cheap comparison baseline | 6 (9) | 10 (15) | 13 (21) |
| Expanded development lexicon | In-sample rule reference | 7 (10) | 12 (18) | 14 (24) |
| Sentence count | Length-only baseline | 6 (8) | 9 (13) | 13 (19) |
These are exploratory results from the data used to develop the feature. Independent annotation and evaluation on a fresh batch are pending. Some correctly retrieved reviews were triggered by sentences about competitors or unrelated matters, so a low-scoring sentence is a ranking signal rather than a verified issue snippet.
The separate Model Sensitivity Inspector in the Space measures score changes after deleting a selected single sentence or contiguous sentence span. Those changes describe model behavior; they do not estimate customer-rating uplift or the effect of an operational intervention.
Evaluation method
I used model revision 2e39ef29d577aa84cfe70596020eb40e708b88a0 with float32 inference, a 512-token limit, and batch size 8. The public CSV is pinned to dataset revision a2aa98f96cc2e829904f47984747667d3be53663; its SHA-256 is ce6d53e65b0f229a81d807aec6e1899dd9fa8f253d8ea215c5b2873affbfb3eb.
Calibration uses five label-stratified folds and seed 20261020. Each review is excluded from fitting its own calibrator and routing thresholds. I minimize NLL with a coordinate/grid search and apply softmax((log(p_raw) + logit_bias) / temperature), fixing the Positive bias to zero for identifiability. Temperature-only calibration fixes all biases to zero; temperature plus bias shares one temperature across classes.
Class-weighted training can shift class probabilities. The inverse-weight comparison uses temperature 1 and bias [-log(3/1.5), -log(8/1.5), 0]. This is a sensitivity baseline motivated by ideal weighted cross-entropy, rather than an assumed correction for this fitted model.
NLL uses natural logs. Multiclass Brier is the mean sum of squared errors across all three classes. ECE uses ten equal-width bins of top-label confidence. Lower values are better for these three metrics.
Limitations
- The 500 reviews follow fixed star quotas drawn from early source test rows, rather than a random sample of the entire split. Their 23% / 10% / 67% class mix is a reference composition, not an estimate of production prevalence or the original 40,000-review class mix. Source test membership alone does not establish independence from my original training and evaluation splits.
- Small slices, especially the 30 two-star reviews, have substantial sampling uncertainty; ECE is also sensitive to sample size and binning. Business, customer, and timestamp identifiers are unavailable, so this benchmark cannot establish generalization across those groups. Cross-fitting reduces calibration leakage but does not turn an exploratory benchmark into an untouched final test.
- The Space uses calibration and thresholds fitted on the full reference set. Future review rates and accuracy can differ, and calibration is not guaranteed to help. Star-level error rates describe the three-class target, not complaint severity.
- Human reviewer accuracy has not been measured. A perfect-reviewer calculation would be a scenario upper bound. This benchmark measures no churn, retention uplift, revenue, or pricing outcomes.
Citation
Developed as part of Georgia Tech OMSA Practicum, Spring 2026 — PathSynch Labs Pod C.