加载中…
加载中…
This model reads the English reviews written by a Steam user and estimates whether that user belongs to the group associated with toxic review behavior. It is the end to end transformer of the paper Toxic Steam: A Large-Scale Characterization and Prediction of Toxic Users in a Gaming Platform, accepted at AIIDE 2026, retrained with the exact recipe of the paper on the same sample of 60,000 users and then evaluated on 60,000 other users that it never saw, at the real platform prevalence of about 4%.
The scores are meant to rank users for human review at platform scale. They are not a verdict on anyone's character and should never trigger an automatic sanction, for the reasons explained in the next section.
Content warning. The training data contains offensive and hateful language.
The study collected 36.8 million English reviews from 14.1 million Steam users through the public Steam Web API, between September 2024 and April 2025. Each review was scored by two automatic detectors, Perspective API and Detoxify, with thresholds calibrated on 400 reviews annotated by three people (Fleiss kappa of 0.85 for Detoxify and 0.75 for Perspective). A user is labeled toxic_user when at least one of their reviews is flagged.
Two consequences follow. The model learns to anticipate the decisions of those detectors, not human judgment, and it reads the same reviews that define the label, so most of its accuracy comes from finding the flagged text itself. In the paper, a leave toxic out control that removes the flagged reviews before representing each user still beats profile metadata (AUC-PR of 0.166 against 0.063 with XGBoost), which shows that a weaker behavioral trace exists beyond the flagged text, but far below the full score. The label also grows with activity, since users who write more reviews have more chances of being flagged.
from transformers import pipeline
clf = pipeline("text-classification", model="vkusterl/toxic-steam-user-distilbert", top_k=None)
def user_text(reviews):
# same input format used in training
return " [SEP] ".join(r[:500] for r in reviews)[:2048]
reviews = [
"Great campaign with friends, terrible matchmaking.",
"Worth it on sale. The developers actually listen to feedback.",
]
scores = {d["label"]: d["score"] for d in clf([user_text(reviews)], truncation=True, max_length=256)[0]}
threshold = clf.model.config.decision_threshold
print(round(scores["toxic_user"], 3), scores["toxic_user"] >= threshold)
Each review is cut at 500 characters, the reviews are joined with [SEP] in order of publication, and the whole text is limited to 2,048 characters and 256 tokens. The threshold stored in the configuration (decision_threshold, 0.927) was chosen on a validation split to maximize macro F1. Whoever uses the scores should move it according to how many cases their reviewers can actually inspect.
All numbers below keep the real prevalence. A random ranking has an AUC-PR equal to the prevalence itself.
Evaluation of this checkpoint on 60,000 users from the same 10% slice of the corpus who were not part of the training sample (2,383 toxic users, prevalence of 3.97%).
| Metric | Value | 95% bootstrap interval |
|---|---|---|
| AUC-PR | 0.648 | 0.630 to 0.668 |
| ROC-AUC | 0.968 | 0.966 to 0.971 |
| F1, toxic class | 0.604 | 0.589 to 0.619 |
| Precision, toxic class | 0.526 | |
| Recall, toxic class | 0.709 | |
| Macro F1 | 0.792 |
For reference, the paper reports the same recipe with ten fold cross validation on the 60,000 user sample, next to the other model families of the benchmark.
| Model | AUC-PR | ROC-AUC | F1, toxic class |
|---|---|---|---|
| Profile metadata, LightGBM | 0.070 | 0.637 | 0.101 |
| Content embeddings, XGBoost | 0.476 | 0.932 | 0.463 |
| DistilBERT end to end, this recipe | 0.643 | 0.968 | 0.605 |
| Claude Opus as a classifier | 0.408 | 0.921 | 0.389 |
| GPT-4o as a classifier | 0.313 | 0.688 | 0.405 |
| Llama 3.1 8B as a classifier | 0.308 | 0.818 | 0.344 |
| Qwen 2.5 7B as a classifier | 0.268 | 0.888 | 0.305 |
The classical models were evaluated with five folds on all 620,891 users of the slice, and the LLMs on only 1,000 users with 40 positives, so the LLM numbers carry much wider intervals. In the paper, the LLMs were more useful as generators of auditable explanations for human reviewers than as classifiers.
Results of this checkpoint by number of English reviews written by the user.
| English reviews by the user | Users | Toxic users | Prevalence | AUC-PR | F1, toxic class |
|---|---|---|---|---|---|
| 1 | 33,947 | 510 | 1.5% | 0.685 | 0.644 |
| 2 to 4 | 19,411 | 905 | 4.7% | 0.680 | 0.616 |
| 5 or more | 6,642 | 968 | 14.6% | 0.602 | 0.567 |
Users with a single review are the easiest group, because their label is decided by the very text the model reads. Prolific users are harder even though they are toxic far more often. Their input is cut at 256 tokens, so the reviews that define the label are often left out, and recall drops accordingly.
| Setting | Value |
|---|---|
| Base model | distilbert/distilbert-base-uncased |
| Data | 60,000 users sampled at real prevalence from a 10% hash slice of 620,891 users with public profiles, 90% for training and 10% to choose the threshold |
| Balancing | Negatives undersampled to 1:1, 4,320 training examples |
| Optimization | AdamW, learning rate 2e-5, batch of 16, 2 epochs (540 steps) |
| Input | Up to 256 tokens per user |
| Hardware | NVIDIA GTX 1070, 7 minutes |
The only change from the paper code is dynamic padding inside each batch, which does not change the computation and makes training faster. The scripts that rebuild this checkpoint and its evaluation are in the scripts folder (preparar.py and treinar_final.py). They need the original corpus, which is not distributed.
The labels come from automatic detectors and inherit their errors. Performative profanity and game vocabulary such as "kill" are known sources of false positives that calibration reduces but does not remove. The corpus is English only, comes from a single collection window and covers the 43.7% of users whose reviews could be matched to a public profile, so behavior in other languages, in chat or in discussion forums is outside its scope. The model classifies users within that window and does not forecast future behavior.
Any use for moderation should keep a person in the loop, report the label definition, the circularity and the uncertainty together with the scores, and treat a positive prediction as a reason to look closer, never as evidence of wrongdoing.
The raw reviews and user data are not distributed. They were collected from the public Steam Web API, are subject to the Steam terms of service, and the people who wrote them did not consent to toxicity profiling. The full pipeline, the result tables and the documentation are in the GitHub repository.
@inproceedings{gibrim2026toxicsteam,
title = {Toxic Steam: A Large-Scale Characterization and Prediction of Toxic Users in a Gaming Platform},
author = {Gibrim, Paula T. M. and Lodi, Vinicius K. and Andrade, Gabriel S. and Ribeiro, Marcus V. G. and Gruppi, Maur{\'i}cio and Barbosa, Daniel M. and Melo, Philipe F. and Reis, Julio C. S.},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE)},
year = {2026}
}
Paula Gibrim and Vinicius Kuster Lodi share the first authorship. This checkpoint was trained and published by Vinicius Kuster Lodi, Universidade Federal de Viçosa.
Modelo DistilBERT que lê as avaliações em inglês de um usuário da Steam e estima se ele pertence ao grupo associado a avaliações tóxicas. É o modelo final do artigo Toxic Steam, aceito no AIIDE 2026, treinado com a mesma receita do artigo e avaliado em 60 mil usuários inéditos na prevalência real de cerca de 4%, com AUC-PR de 0.648 contra 0,07 dos metadados de perfil. O rótulo vem de detectores automáticos e é em grande parte circular em relação ao texto lido pelo modelo, por isso o uso indicado é a triagem para revisão humana, nunca uma decisão automática.