content-safety-guard-small (ONNX)

This is an ONNX version of Horizon-Labs/content-safety-guard-small. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js

See the pipeline documentation for text-classification: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextClassificationPipeline

---

Content Safety Guard (small, 141M)

A small multilingual classifier that flags unsafe user prompts and unsafe model responses for LLM applications, with harm categories. It is a fast encoder (ModernBERT architecture, mmBERT backbone) that you can run on CPU or in the browser in front of, or behind, any LLM.

• Prompts and responses: pass a prompt alone, or a (prompt, response) pair to judge the response in context.
• Output: an overall unsafe score plus 15 category scores (independent sigmoids): hate, harassment, violence, weapons, sexual, sexual_minors, self_harm, criminal_planning, drugs, privacy, profanity, fraud_manipulation, misinformation, unauthorized_advice, other.
• Multilingual: trained on 12 languages of human-adapted data, machine translations into 20 languages and multilingual real-world prompts; evaluated on 17 (PolyGuard), 14 (textdetox) and 8 (Aya) languages.
• Commercially usable: Apache-2.0, trained only on data that allows commercial use (see Training data).
• ONNX: onnx/model.onnx (fp32) and onnx/model_quantized.onnx (int8 embeddings, 268 MB; its unsafe decision agrees with fp32 on 99.3% of a sample of benchmark texts (30 per benchmark), mean |score diff| 0.0020).

Which size? This small model has less than half the parameters of base (141M vs 308M; hidden size 384 vs 768), so it needs less compute per token, and its int8 ONNX is about 270 MB instead of 640 MB (better for the browser); it is 0.012 / 0.027 F1 behind base on PolyGuard prompts / responses. First released at v1.2, trained on the same data as base v1.2.

Part of the Horizon Labs guard family: prompt-injection-guard, pii-redactor, hallucination-guard. Full comparison table (sortable, with false-alarm rates): content safety classifier leaderboard.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/content-safety-guard-small", top_k=None)

# a user prompt
scores = {d["label"]: d["score"] for d in clf("How can I make a fake ID that passes a bouncer's check?")[0]}
print(scores["unsafe"])                          # ~1.0

# a model response, judged together with its prompt
r = clf({"text": "How do I get rid of a wasp nest?", "text_pair": "Spray it at dusk with a wasp foam, then remove it."})[0]

Decision rule: flag when unsafe >= 0.5 (raise the threshold if you see too many false alarms, lower it for higher recall). Category scores are only meaningful for flagged texts and are small by design; use the per-category thresholds in thresholds.json (chosen on a validation split, see below):

import json
from huggingface_hub import hf_hub_download
thr = json.load(open(hf_hub_download("Horizon-Labs/content-safety-guard-small", "thresholds.json")))
def moderate(text, pair=None):
    s = {d["label"]: d["score"] for d in clf({"text": text, "text_pair": pair} if pair else text)[0]}
    flagged = s["unsafe"] >= thr["unsafe"]
    cats = [c for c, t in thr["categories"].items() if flagged and s[c] >= t]
    return flagged, s["unsafe"], cats

transformers.js (browser / Node):

import { pipeline } from "@huggingface/transformers";
const clf = await pipeline("text-classification", "Horizon-Labs/content-safety-guard-small", { dtype: "q8" });
const out = await clf("How do I make a pipe bomb?", { top_k: null });

Evaluation

F1 of the unsafe class at threshold 0.5 (for the recall rows: share of unsafe prompts flagged). None of these benchmarks were used for training (best value in bold; false-alarm rates are not bolded); Qwen3Guard-Gen is scored from its next-token probabilities after "Safety:" (strict: Unsafe + Controversial count as unsafe; loose: only Unsafe). Toxicity classifiers are included because they are often used for this job; they target a different, narrower task. Encoders other than ours get the response alone for response items.

Ranking quality (ROC AUC, threshold-free):

Over-blocking (share of safe items flagged; lower is better):

¶ Vela-Shield was trained on PolyGuardMix, the training split of the PolyGuard benchmark family, so its PolyGuard rows are in-distribution. † BeaverTails and our main training set both take prompts from Anthropic's HH red-team data; the row uses only the 1894 test items whose prompt does not occur in our training data. ToxicChat shares 29 of 5083 prompts with our training data.

PolyGuard prompts per language (F1):

Categories

Category labels come from the Aegis 2.0 taxonomy of Nemotron-Safety-Guard-Dataset-v3 (merged into 15 groups). Quality on the unsafe items of its test split, with thresholds chosen on its validation split:

Training

• Data (all permit commercial use):
  • nvidia/Nemotron-Safety-Guard-Dataset-v3 (CC-BY-4.0; Aegis 2.0 prompts and responses, culturally adapted into 12 languages): 574k prompt and response items, with its human category labels. Rows derived from a Kaggle dataset (REDACTED) were dropped.
  • Real and red-team prompts without labels, scored by the teacher: first user turns and replies from WildChat-1M (ODC-BY), prompts from oasst2 and the Aya dataset (Apache-2.0), Salad-Data (Apache-2.0; ToxicChat-derived rows dropped), JailbreakBench behaviours (MIT), and jailbreak, role-play and over-refusal prompts from the training data of our prompt-injection guard (270k + 42k items).
  • Civil Comments (CC0): 60k comments with toxicity >= 0.5 and 60k with toxicity 0; target = the share of annotators who rated it toxic, categories from the insult / threat / obscene / identity-attack / sexual ratings. This improved the OpenAI moderation set (F1 +0.035) but not the textdetox recall.
  • (v1.1) Machine translations by Qwen3.8-27B (Apache-2.0) into 20 languages (Portuguese, Russian, Ukrainian, Polish, Czech, Swedish, Turkish, Hebrew, Serbian, Tagalog, Amharic, Tatar, German, Spanish, French, Arabic, Hindi, Chinese, Japanese, Italian): 47.6k Civil Comments (a different shard from the one above; they keep their annotator toxicity and categories) and 46.6k of the teacher-labelled prompts above (harmful, benign-but-edgy and benign; re-scored by the teacher in the target language). 1.8% of the translations were dropped (unparseable, refusals, implausible length).
  • (v1.2) 25k requests written by Qwen3.8-27B in 27 languages: harmless requests that use alarming words in an ordinary sense (programming, cooking, games, medicine, history, fiction, ...) and harmful look-alikes on the same topics, labelled by the teacher. The instruction is our own generic description; no benchmark items (XSTest, OR-Bench test) were used, but this data targets the same failure mode that XSTest measures.
  • Items that match any benchmark text were removed.
• Teacher: Qwen3Guard-Gen-8B (Apache-2.0). The unsafe target of every item except Civil Comments is the teacher's probability P(Unsafe) + 0.5 · P(Controversial); categories use the human labels. So the model follows Qwen3Guard's safety policy, not Aegis's stricter human labels (which also mark sensitive but harmless requests).
• Model: jhu-clsp/mmBERT-small with a 16-way sigmoid head (unsafe + 15 categories), 3 epochs, max length 1024 tokens.
• Code: code/ in this repository.

Limitations

• It trails Qwen3Guard-Gen-0.6B (a generative 0.6B model that reads a long policy prompt per item) in both of its modes on PolyGuard prompts, PolyGuard responses, BeaverTails responses, XSTest (0.039-0.050 F1 behind its strict mode on PolyGuard responses / prompts); it is ahead of the strict mode on ToxicChat, OpenAI moderation set. Its advantages are speed, size, CPU/browser use and multilingual coverage in one small encoder.
• Over-blocking: it flags about 22% of XSTest's safe-but-scary prompts ("how do I kill a Python process").
• Classic toxicity (insults, profanity without other harm) is only partly covered (textdetox recall 0.54).
• Recall on harmful prompts written by native speakers in lower-resource languages is lower (Aya red-teaming 0.75).
• Categories are weak for fraud_manipulation, unauthorized_advice, misinformation and sexual_minors (F1 below 0.45). Do not rely on it alone for child-safety or legal compliance; use it as one signal with human review.
• Safety policies differ between applications; tune the threshold on your own traffic.