DistilBERT toxicity classifier (soft labels + identity weights)

Fine-tuned distilbert-base-uncased that predicts whether an English comment is toxic. Built for the capstone project Trust in Automated Moderation of Toxic Comments (ECE 4424 / CS 4824). Code: https://github.com/martinagg7/ML-ToxicModeration

Training

Data: Jigsaw Unintended Bias in Toxicity Classification (train.csv, 1.8 M news comments), 80/10/10 split grouped by article.
Soft labels: the model learns the share of annotators who marked a comment as toxic, not only 0/1.
Identity weights: comments that mention an identity weigh more in the loss (2 if toxic, 3 if non-toxic).
2 epochs, learning rate 2e-5, batch size 16, max length 128 tokens. Epoch chosen on validation PR-AUC.

Results (held-out test set, 177,635 comments, threshold 0.5)

Model	Precision	Recall	F1	ROC AUC	PR-AUC
Word blocklist (LDNOOBW)	0.27	0.10	0.15	0.54	0.10
TF-IDF + logistic regression	0.73	0.53	0.61	0.93	0.69
This model	0.80	0.58	0.67	0.96	0.78

Limitations

Misses about 4 in 10 toxic comments at threshold 0.5.
Still blocks more innocent comments when they mention an identity (for example 5.6% for gay/lesbian vs 1.3% overall).
Fails on negations with insults ("You are not an idiot"), counter-speech and positive swearing.
Evaluated only on English news comments from the same source as training.

Not intended for fully automatic moderation: uncertain predictions (probability 0.4–0.6) should go to human review.

Usage

from transformers import pipeline
clf = pipeline("text-classification", model="Martinagg/toxic-distilbert")
clf("You are an idiot")

Loading…