Loading…
Loading…
Content warning: trained on real toxic comments. Portfolio demo, not a production moderation system.
A portfolio project by Malav Naik. Code: github.com/malavnaik99/moderation-classifier · Notebook: Kaggle · Demo: Space
What it does: gives a 0–1 score for five categories: toxicity, obscene, identity_attack, insult, threat (in that output order; sigmoid per output, multi-label).
Training: distilbert-base-uncased, 1 epoch on a random 300,000 comments from the training split of the Jigsaw Unintended Bias / Civil Comments data (CC0), max 128 tokens, Kaggle free GPU. A label counts as positive when ≥ 50% of raters said yes. severe_toxicity was left out: too rare to measure.
Intended use: learning and demonstration of moderation metrics. Out of scope: making real decisions about people or content.
| Toxicity | TF-IDF + logistic regression baseline | This model |
|---|---|---|
| ROC-AUC | 0.941 | 0.957 |
| PR-AUC | 0.706 | 0.755 |
| Recall at ≥ 90% precision ("auto-action" policy) | 30.9% | 36.6% |
| Share of comments sent to human review to catch ~80% of toxic comments | 10.7% | 8.5% |
Per-category ROC-AUC: obscene 0.979, identity attack 0.978, insult 0.968, threat 0.973.
Thresholds (chosen on the validation set): auto-action ≥ 0.821, send to review ≥ 0.219 (thresholds.json).
Share of non-toxic comments flagged for review, on test comments with identity annotations.
| Comment mentions | Baseline | This model |
|---|---|---|
| (all identity-annotated comments) | 12.8% | 9.6% |
| black | 49.5% | 35.6% |
| white | 44.3% | 31.0% |
| homosexual gay or lesbian | 38.6% | 29.9% |
| muslim | 37.2% | 25.8% |
| transgender | 31.6% | 24.5% |
| psychiatric or mental illness | 29.5% | 24.0% |
| jewish | 23.8% | 16.0% |
| male | 20.9% | 14.6% |
| female | 19.7% | 14.0% |
| asian | 14.2% | 8.8% |
| christian | 12.4% | 8.2% |
Limitation: harmless comments that mention some identities are still flagged up to 3.7× more often than average. Do not use it to make decisions about people or content without human review.
The demo runs a compressed copy (onnx/model_quantized.onnx, int8) in the visitor's browser. On 2,000 random Civil Comments test comments it gave the same publish / review / auto-action decision as the full model 97.2% of the time; toxicity scores differed by 0.015 on average (largest difference 0.28).
distilbert/distilbert-base-uncased