DistilBERT — multi-label content / policy classification

Fine-tuned distilbert-base-uncased predicting all applicable labels for a comment: toxic, severe_toxic, obscene, threat, insult, identity_hate.

Six independent sigmoid outputs trained with BCEWithLogitsLoss — not softmax — so any combination of labels can fire and the probabilities do not sum to 1.

Metrics (held-out test split, n=23,936)

Macro-F1: 0.6836 · micro-F1: 0.7875 · macro average precision: 0.7359

Why not accuracy

A model that predicts nothing scores 89.83% exact-match accuracy and 96.33% mean per-label accuracy on this data, at macro-F1 of exactly 0.0. Macro-F1 is the headline for that reason.

Thresholds

The per-label thresholds in the table were calibrated on the validation split and applied unchanged to test. thresholds_transformer.json in this repo holds them. Using them instead of a flat 0.5 is worth +0.021 macro-F1 with no retraining.

Limitations

• Trained on 2018 English Wikipedia talk-page comments; other domains will degrade.
• Rare labels are weak: F1 ~0.54 on threat and severe_toxic; identity_hate recall 0.535.
• threat's threshold rests on only 72 validation positives, so it is a noisy estimate.
• Inputs truncated at 256 tokens (~6.9% of training comments were longer).
• A demo, not a moderation system. Do not make consequential decisions about people with it.

Code, full write-up, and a live demo: https://github.com/Srijan-Ratrey/Multilabel-content-classification